Method and apparatus for obtaining panoramic and rectilinear images using rotationally symmetric wide-angle lens

ABSTRACT

The present invention provides mathematically accurate image processing algorithms for extracting natural looking panoramic images and distortion-free rectilinear images from images acquired using a camera equipped with a wide-angle lens which is rotationally symmetric about an optical axis and devices implementing such algorithms. Imaging systems using this method can be used not only in security•surveillance applications for indoor and outdoor environments, but also in diverse areas such as video phone for apartment entrance door, rear view camera for vehicles, visual sensor for unmanned aerial vehicles and robots, camera phone, PC camera, and broadcasting camera. Also, it can be used to obtain panoramic or rectilinear photographs using a digital camera.

TECHNICAL FIELD

The present invention generally relates to mathematically precise imageprocessing methods of extracting distortion-free rectilinear images andpanoramic images, which appear most natural to the naked eye, fromimages acquired using a camera equipped with a wide-angle lens that isrotationally symmetric about an optical axis, as well as devices usingthe methods.

BACKGROUND ART

Panoramic camera, which captures the 360° view of scenic places such astourist resorts, is an example of a panoramic imaging system. Panoramicimaging system is an imaging system that captures the views one couldget by making one complete turn-around from a given spot. On the otherhand, omnidirectional imaging system captures the view of every possibledirection from a given position. Omnidirectional imaging system providesa view that a person could observe from a given position by turningaround as well as looking up and down. In a mathematical terminology,the solid angle of the region that can be captured by the imaging systemis 4π steradian.

There have been a lot of studies and developments of panoramic imagingsystems not only in the traditional areas such as photographingbuildings, nature scenes, and heavenly bodies, but also insecurity/surveillance systems using CCD (charge-coupled device) or CMOS(complementary metal-oxide-semiconductor) cameras, virtual touring ofreal estates, hotels and tourist resort, and navigational aids formobile robots and unmanned aerial vehicles (UAV).

One method of obtaining a panoramic image is to employ a fisheye lenswith a wide field of view (FOV). For example, the entire sky and thehorizon can be captured in a single image by pointing a camera equippedwith a fisheye lens with 180° FOV toward the zenith (i.e., the opticalaxis of the camera is aligned perpendicular to the ground plane). Onthis reason, fisheye lenses have been often referred to as “all-skylenses”. Particularly, a high-end fisheye lens by Nikon, namely, 6 mmf/5.6 Fisheye-Nikkor, has a FOV of 220°. Therefore, a camera equippedwith this lens can capture a portion of the backside of the camera aswell as the front side of the camera. Then, panoramic image can beobtained from thus obtained fisheye image after proper image processing.

In many cases, imaging system is installed on vertical walls. Imagingsystems installed on outside walls of a building for the purpose ofmonitoring the surroundings, or a rear view camera for monitoring thebackside of a passenger car are such examples. In such cases, it isinefficient if the horizontal field of view is significantly larger than180°. This is because a wall, which is not needed to be monitored, takesup a large space in the monitor screen. Pixels are wasted in this case,and screen appears dull. Therefore, a horizontal FOV around 180° is moreappropriate for such cases. Nevertheless, a fisheye lens with 180° FOVis not desirable for such application. This is because the barreldistortion, which accompanies a fisheye lens, evokes psychologicaldiscomfort and abhorred by the consumer.

An example of an imaging system, which can be installed on an interiorwall for the purpose of monitoring the entire room, is given by apan•tilt•zoom camera. Such a camera is comprised of a video camera,which is equipped with an optical zoom lens, mounted on a pan•tiltstage. Pan is an operation of rotating in the horizontal direction for agiven angle, and tilt is an operation of rotating in the verticaldirection for a given angle. In other words, if we assume that thecamera is at the center of a celestial sphere, then pan is an operationof changing the longitude, and tilt is an operation of changing thelatitude. Therefore, the theoretical range of pan operation is 360°, andthe theoretical range of tilt operation is 180°. The shortcomings of apan•tilt•zoom camera include high price, large size and heavy weight.Optical zoom lens is large, heavy and expensive due to the difficulty indesign and the complicated structure. Also, a pan•tilt stage is anexpensive device no cheaper than a camera. Therefore, it cost aconsiderable sum of money to install a pan•tilt•zoom camera.Furthermore, since a pan•tilt•zoom camera is large and heavy, this pointcan become a serious impediment to certain applications. Examples ofsuch cases include airplanes where the weight of the payload is ofcritical importance, or a strict size limitation exists in order toinstall a camera in a confined space. Furthermore, pan•tilt•zoomoperation takes a time because it is a mechanical operation. Therefore,depending on the particular application at hand, such a mechanicaloperation may not be fast enough.

References 1 and 2 provide fundamental technologies of extracting animage having a particular viewpoint or projection scheme from an imagehaving other than the desirable viewpoint or projection scheme.Specifically, reference 2 provides an example of a cubic panorama. Inshort, a cubic panorama is a special technique of illustration whereinthe observer is assumed to be located at the very center of an imaginarycubic room made of glass, and the outside view from the center of theglass room is directly transcribed on the region of the glass wall wherethe ray vector from the object to the observer meets the glass wall. Anexample of a more advanced technology is provided in the above referencewith which reflections from an arbitrarily shaped mirrored surface canbe calculated. Specifically, the author of reference 2 created animaginary lizard having a highly reflective mirror-like skin as if madeof a metal surface, then set-up an observer's viewpoint separated fromthe lizard, and calculated the view of the imaginary environmentreflected on the lizard skin from the viewpoint of the imaginaryobserver. However, the environment was not a real environment capturedby an optical lens, but a computer-created imaginary environmentcaptured with an imaginary distortion-free pinhole camera.

On the other hand, an imaging system is described in reference 3 that isable to perform pan•tilt•zoom operations without a physically movingpart. The said invention uses a camera equipped with a fisheye lens withmore than 180° FOV in order to take a picture of the environment. Then,user designates a principal direction of vision using various devicessuch as a joystick, upon which, the computer extracts a rectilinearimage from the fisheye image that could be obtained by heading adistortion-free camera to that particular direction. The main differencebetween this invention and the prior arts is that this invention createsa rectilinear image corresponding to the particular direction the userhas designated using devices such as a joystick or a computer mouse.Such a technology is a core technology in the field of virtual reality,or when it is desirable to replace mechanical pan•tilt•zoom camera, andthe keyword is “interactive picture”. In this technology, there are nophysically moving parts in the camera. As a consequence, the systemresponse is fast, and there is less chance of mechanical failure.

Ordinarily, when an imaging system such as a security camera isinstalled, a cautionary measure is taken so that a vertical lineperpendicular to the horizontal plane also appears vertical in theacquired image. In such a case, vertical lines still appear verticaleven as mechanical pan•tilt•zoom operation is performed. On the otherhand, in the said invention, vertical lines generally do not appear asvertical lines after a software pan•tilt•zoom operation has beenperformed. To remedy such an unnatural result, a rotate operation isadditionally performed, which is not found in a mechanical pan•tilt•zoomcamera. Furthermore, the said invention does not provide the exactamount of rotate angle, which is needed in order to display verticallines as vertical lines. Therefore, the rotation angle must be found ina trial-and-error method in order to display vertical lines as verticallines.

Furthermore, the said invention assumes that the projection scheme ofthe fisheye lens is an ideal equidistance projection scheme. But, thereal projection scheme of a fisheye lens generally shows a considerabledeviation from an ideal equidistance projection scheme. Since the saidinvention does not take into account the distortion characteristics ofthe real lens, images obtained after image processing still showsdistortion.

The invention described in reference 4 remedies the shortcoming of theinvention described in reference 3, namely the inability of taking intoaccount the real projection scheme of the fisheye lens used in imageprocessing. Nevertheless, the defect of not showing vertical lines asvertical lines in the monitor screen has not been resolved.

From another point of view, all animals and plants including human arebound on the surface of the earth due to the gravitational pull, andmost of the events, which need attention or cautionary measure, takeplace near the horizon. Therefore, even though it is necessary tomonitor every 360° direction on the horizon, it is not as important tomonitor high along the vertical direction, for example, as high as tothe zenith or deep down to the nadir. Distortion is unavoidable if wewant to describe the scene of every 360° direction on a two-dimensionalplane. Similar difficulty exists in the cartography where geography onearth, which is a structure on the surface of a sphere, needs to bemapped on a planar two-dimensional atlas. Among all the distortions, thedistortion that appears most unnatural to the people is the distortionwhere vertical lines appear as curved lines. Therefore, even if otherkinds of distortions are present, it is important to make sure that sucha distortion is absent.

Described in reference 5 are the well-known map projection schemes amongthe diverse map projection schemes such as equi-rectangular projection,Mercator projection and cylindrical projection schemes, and reference 6provides a brief history of diverse map projection schemes. Among these,the equi-rectangular projection scheme is the projection scheme mostfamiliar to us when we describe the geography on the earth, or when wedraw the celestial sphere in order to make a map of the constellation.

Referring to FIG. 1, if we assume the surface of the earth or thecelestial sphere is a spherical surface with a radius S, then anarbitrary point Q on the earth's surface has a longitude ψ and alatitude δ. On the other hand, FIG. 2 is a schematic diagram of a planarmap drawn according to the equi-rectangular projection scheme. A point Qon the earth's surface having a longitude ψ and a latitude δ has acorresponding point P″ on the planar map(234) drawn according to theequi-rectangular projection scheme. The rectangular coordinate of thiscorresponding point is given as (x″, y″). Furthermore, the referencepoint on the equator having a longitude Wand a latitude 0° has acorresponding point O″ on the planar map, and this corresponding pointO″ is the origin of the rectangular coordinate system. Here, accordingto the equi-rectangular projection scheme, the same interval in thelongitude (i.e., the same angular distance along the equator)corresponds to the same lateral interval on the planar map. In otherwords, the lateral coordinate x″ on the planar map(234) is proportionalto the longitude.

Math Figure 1

x″=cψ  [Math.1]

Here, c is proportionality constant. Also, the longitudinal coordinatey″ is proportional to the latitude, and has the same proportionalityconstant as the lateral coordinate.

Math Figure 2

y″=cδ  [Math.2]

The span of the longitude is 360° ranging from −180° to +180°, and thespan of the latitude is 180° ranging from −90° to +90°. Therefore, a mapdrawn according to the equi-rectangular projection scheme must have awidth W:height ratio of 360:180=2:1.

Furthermore, if the proportionality constant c is given as the radius Sof the earth, then the width of the said planar map is given as theperimeter of the earth measured along the equator as given in Eq. 3.

Math Figure 3

W=2πS  [Math.3]

Such an equi-rectangular projection scheme appears as a naturalprojection scheme considering the fact that the earth's surface is closeto the surface of a sphere. Nevertheless, it is disadvantageous in thatthe size of a geographical area is greatly distorted. For example, twovery close points near the North Pole can appear as if they are on theopposite sides of the earth in a map drawn according to theequi-rectangular projection scheme.

n the other hand, in a map drawn according to the Mercator projectionscheme, the longitudinal coordinate is given as a complex function givenin Eq. 4.

Math Figure 4

$\begin{matrix}{y^{''} = {c\; \ln \left\{ {\tan \left( {\frac{\pi}{4} + \frac{\delta}{2}} \right)} \right\}}} & \left\lbrack {{Math}.\mspace{14mu} 4} \right\rbrack\end{matrix}$

On the other hand, FIG. 3 is a conceptual drawing of a cylindricalprojection scheme or a panoramic perspective. In a cylindricalprojection scheme, an imaginary observer is located at the center N of acelestial sphere(331) with a radius S, and it is desired to make a mapof the celestial sphere centered on the observer, the map covering mostof the region excluding the zenith and the nadir. In other words, thespan of the longitude must be 360° ranging from −180° to +180°, but therange of the latitude can be narrower including the equator within itsspan. Specifically, the span of the latitude can be assumed as rangingfrom −Δ to +Δ, and here, Δ must be smaller than 90°.

In this projection scheme, a hypothetical cylindrical plane(334) isassumed which contacts the celestial sphere(331) at the equator(303).Then, for a point Q(ψ, δ) on the celestial sphere having a givenlongitude ψ and a latitude δ, a line segment connecting the center ofthe celestial sphere and the point Q is extended until it meets the saidcylindrical plane. This intersection point is designated as P(ψ, δ). Inthis manner, the corresponding point P on the cylindrical plane(334) canbe obtained for every point Q on the celestial sphere(331) within thesaid latitude range. Then, a map having a cylindrical projection schemeis obtained when the cylindrical plane is cut and flattened out.Therefore, the lateral coordinate x″ of the point P on the flattened-outcylindrical plane is given by Eq. 5, and the longitudinal coordinate y″is given by Eq. 6.

Math Figure 5

x″=Sψ  [Math.5]

Math Figure 6

y″=S tan δ  [Math.6]

Such a cylindrical projection scheme is the natural projection schemefor a panoramic camera that produces a panoramic image by rotating inthe horizontal plane. Especially, if the lens mounted on the rotatingpanoramic camera is a distortion-free rectilinear lens, then theresulting panoramic image exactly follows a cylindrical projectionscheme. In principle, such a cylindrical projection scheme is the mostaccurate panoramic projection scheme. However, the panoramic imageappears unnatural when the latitude range is large, and thus it is notwidely used in practice. Unwrapped panoramic image thus produced andhaving a cylindrical projection scheme has a lateral width W given byEq. 3. On the other hand, if the range of the latitude is from δ₁ to δ₂,then the longitudinal height of the unwrapped panoramic image is givenby Eq. 7.

Math Figure 7

H=S(tan δ₂−tan δ₁)  [Math.7]

Therefore, the following equation can be derived from Eq. 3 and Eq. 7.

Math Figure 8

$\begin{matrix}{\frac{W}{H} = \frac{2\pi}{{\tan \; \delta_{2}} - {\tan \; \delta_{1}}}} & \left\lbrack {{Math}.\mspace{14mu} 8} \right\rbrack\end{matrix}$

Therefore, an unwrapped panoramic image following a cylindricalprojection scheme must satisfy Eq. 8.

All the animals, plants and inanimate objects such as buildings on theearth are under the influence of gravity, and the direction ofgravitational force is the up-right direction or the vertical direction.Ground plane is fairly perpendicular to the gravitational force, butneedless to say, it is not so on a slanted ground. Therefore, the word“ground plane” actually refers to the horizontal plane, and the verticaldirection is the direction perpendicular to the horizontal plane.Therefore, even if we refer them as the ground plane, the lateraldirection, and the longitudinal direction, for the sake of simplicity,the ground plane must be understood as the horizontal plane, thevertical direction must be understood as the direction perpendicular tothe horizontal plane, and the horizontal direction must be understood asa direction parallel to the horizontal plane, whenever an exact meaningof a term needs to be clarified.

Panoramic lenses described in references 7 and 8 take panoramic imagesin one shot with the optical axis of the panoramic lens aligned verticalto the ground plane. A cheaper alternative to the panoramic imageacquisition method by the previously described camera with ahorizontally-rotating lens consist of taking an image with an ordinarycamera with the optical axis horizontally aligned, and repeating to takepictures after horizontally rotating the optical axis by a certainamount. Four to eight pictures are taken in this way, and a panoramicimage with a cylindrical projection scheme can be obtained by seamlesslyjoining the pictures consecutively. Such a technique is calledstitching. QuickTime VR from Apple computer inc. is commercial softwaresupporting this stitching technology. This method requires a complex,time-consuming, and elaborate operation of precisely joining severalpictures and correcting the lens distortion.

According to the reference 9, another method of obtaining a panoramic oran omnidirectional image is to take a hemispherical image byhorizontally pointing a camera equipped with a fisheye lens with morethan 180° FOV, and then point the camera to the exact opposite directionand take another hemispherical image. By stitching the two imagesacquired from the camera using appropriate software, one omnidirectionalimage having the view of every direction (i.e., 4π steradian) can beobtained. By sending thus obtained image to a geographically separatedremote user using communication means such as the Internet, the user canselect his own viewpoint from the received omnidirectional imageaccording to his own personal interest, and image processing software onthe user's computing device can extract a partial image corresponding tothe user-selected viewpoint, and a perspectively correct planar imagecan be displayed on the computing device. Therefore, through the imageprocessing software, the user can make a choice of turning around (pan),looking-up or down (tilt), or take a close (zoom in) or remote (zoomout) view as if the user is actually at the specific place in the image.This method has a distinctive advantage of multiple users accessing thesame Internet site to be able to take looks along the directions oftheir own choices. This advantage cannot be enjoyed in a panoramicimaging system employing a motion camera such as a pan•tilt camera.

References 10 and 11 describe a method of obtaining an omnidirectionalimage providing the views of every direction centered on the observer.Despite the lengthy description of the invention, however, theprojection scheme provided by the said references is one kind ofequidistance projection schemes in essence. In other words, thetechniques described in the documents make it possible to obtainomnidirectional images from a real environment or from a cubic panorama,but the obtained omnidirectional image follows an equidistanceprojection scheme only and its usefulness is thus limited.

On the other hand, reference 12 provides an algorithm for projecting anOmnimax movie on a semi-cylindrical screen using a fisheye lens.Especially, taking into account of the fact that the projection schemeof the fisheye lens mounted on a movie projector deviates from an idealequidistance projection scheme, a method is described for locating theposition of the object point on the film corresponding to a certainpoint on the screen whereon an image point is formed. Therefore, it ispossible to calculate what image has to be on the film in order toproject a particular image on the screen, and such an image on the filmis produced using a computer. Especially, since the lens distortion isalready reflected in the image processing algorithm, a spectator nearthe movie projector can entertain himself with a satisfactory panoramicimage. Nevertheless, the real projection scheme of the fisheye lens inthe said reference is inconvenient to use because it has been modeledwith the real image height on the film plane as the independentvariable, and the zenith angle of the incident ray as the dependentvariable. Furthermore, unnecessarily, the real projection scheme of thefisheye lens has been modeled only with an odd polynomial.

Reference 13 provides examples of stereo panoramic images produced byProfessor Paul Bourke. Each of the panoramic images follows acylindrical projection scheme, and a panoramic image of an imaginaryscene produced by a computer as well as a panoramic image produced by arotating slit camera are presented. For panoramic images produced by acomputer or produced by a traditional method of rotating slit camera,the lens distortion is not an important issue. However, rotating slitcamera cannot be used to take a real-time panoramic image (i.e., movie)of a real world.

References 14 and 15 provide an example of a fisheye lens with 190° FOV,and reference 16 provides various examples of wide-angle lensesincluding dioptric and catadioptric fisheye lenses with stereographicprojection schemes.

On the other hand, reference 17 provides various examples of obtainingpanoramic images following cylindrical projection schemes,equi-rectangular projection schemes, and Mercator projection schemesfrom images acquired using rotationally symmetric wide-angle lensesincluding fisheye lenses. Referring to FIG. 4 through FIG. 12, most ofthe examples provided in the said reference can be summarized asfollows.

FIG. 4 is a conceptual drawing illustrating the real projection schemesof rotationally symmetric wide-angle lenses(412) including fisheyelenses. Z-axis of the world coordinate system describing objectscaptured by the wide-angle lens coincides with the optical axis(401) ofthe wide-angle lens(412). An incident ray(405) having a zenith angle θwith respect to the Z-axis is refracted by the lens(412), and as arefracted ray(406), converges toward an image point P on the focalplane(432). The distance between the nodal point N of the lens and thesaid focal plane is approximately equal to the effective focal length ofthe lens. The sub area on the focal plane whereon real image points havebeen formed is the image plane(433). To obtain a sharp image, the saidimage plane(433) must coincide with the image sensor plane(413) withinthe camera body(414). Said focal plane and the said image sensor planeare perpendicular to the optical axis. The intersection point O betweenthe optical axis(401) and the image plane(433) is hereinafter referredto as the first intersection point. The distance between the firstintersection point and the said image point P is r.

For general wide-angle lenses, the image height r is given by Eq. 9.

Math Figure 9

r=r(θ)  [Math.9]

Here, the unit of the incidence angle θ is radian, and the abovefunction r(θ) is a monotonically increasing function of the zenith angleθ of the incident ray.

Such a real projection scheme of a lens can be experimentally measuredusing an actual lens, or can be calculated from the lens prescriptionusing dedicated lens design software such as Code V or Zemax. Forexample, the y-axis coordinate y of the image point on the focal planeby an incident ray having given horizontal and vertical incidence anglescan be calculated using a Zemax operator REAY, and the x-axis coordinatex can be similarly calculated using an operator REAX.

FIG. 5 is an imaginary interior scene produced by professor Paul Bourkeby using a computer, and it has been assumed that the imaginary lensused to capture the image is a fisheye lens with 180° FOV having anideal equidistance projection scheme.

This image is a square image, of which both the lateral and thelongitudinal dimensions are 250 pixels. Therefore, the coordinate of theoptical axis is (125.5, 125.5), and the image height for an incident raywith zenith angle 909 s given as r′(π/2)=125.5−1=124.5. Here, r′ is nota physical distance, but an image height measured in pixel distance.Since this imaginary fisheye lens follows an equidistance projectionscheme, the projection scheme of this lens is given by Eq. 10.

Math Figure 10

$\begin{matrix}{{r^{\prime}(\theta)} = {{\frac{124.5}{\left( \frac{\pi}{2} \right)}\theta} = {79.26\theta}}} & \left\lbrack {{Math}.\mspace{14mu} 10} \right\rbrack\end{matrix}$

FIG. 6 through FIG. 8 show several embodiments of wide-angle lensespresented in reference 16. FIG. 6 is a dioptric (i.e., refractive)fisheye lens with a stereographic projection scheme, FIG. 7 is acatadioptric fisheye lens with a stereographic projection scheme, andFIG. 8 is a catadioptric panoramic lens with a rectilinear projectionscheme. In this manner, the wide-angle lenses from the said referenceand in the current invention are not limited to a fisheye lens with anequidistance projection scheme, but encompass all kind of wide-anglelenses that are rotationally symmetric about the optical axes.

The main point of the invention in reference 17 is about methods ofobtaining panoramic images by applying mathematically accurate imageprocessing algorithms on images obtained using rotationally symmetricwide-angle lenses. Numerous embodiments in reference 17 can besummarized as follows.

The world coordinate system of the said invention takes the nodal pointN of a rotationally symmetric wide-angle lens as the origin, and avertical line passing through the origin as the Y-axis. Here, thevertical line is a line perpendicular to the ground plane, or moreprecisely to the horizontal plane(917). The X-axis and the Z-axis of theworld coordinate system are contained within the ground plane. Theoptical axis(901) of the said wide-angle lens generally does notcoincide with the Y-axis, and can be contained within the ground plane(i.e., parallel to the ground), or is not contained within the groundplane. The plane(904) containing both the said Y-axis and the saidoptical axis(901) is referred to as the reference plane. Theintersection line(902) between this reference plane(904) and the groundplane(917) coincides with the Z-axis of the world coordinate system. Onthe other hand, an incident ray(905) originating from an object point Qhaving a rectangular coordinate (X, Y, Z) in the world coordinate systemhas an altitude angle 6 from the ground plane, and an azimuth angle ψwith respect to the reference plane. The plane(906) containing both theY-axis and the said incident ray(905) is the incidence plane. Thehorizontal incidence angle ψ of the said incident ray with respect tothe said reference plane is given by Eq. 11.

Math Figure 11

$\begin{matrix}{\psi = {\tan^{- 1}\left( \frac{X}{Z} \right)}} & \left\lbrack {{Math}.\mspace{14mu} 11} \right\rbrack\end{matrix}$

On the other hand, the vertical incidence angle (i.e., the altitudeangle) δ subtended by the said incident ray and the X-Z plane is givenby Eq. 12.

Math Figure 12

$\begin{matrix}{\delta = {\tan^{- 1}\left( \frac{Y}{\sqrt{X^{2} + Z^{2}}} \right)}} & \left\lbrack {{Math}.\mspace{14mu} 12} \right\rbrack\end{matrix}$

The elevation angle μ of the said incident ray is given by Eq. 13,wherein x is an arbitrary angle larger than −90° and smaller than 90°.

Math Figure 13

μ=δ−χ  [Math.13]

FIG. 10 is a schematic diagram of a device of the current invention,which also coincides with that of the reference 17, having an imagingsystem which mainly includes an image acquisition means(1010), an imageprocessing means(1016) and image display means(1015, 1017). The imageacquisition means(1010) includes a rotationally symmetric wide-anglelens(1012) and a camera body(1014) having an image sensor(1013) inside.The said wide-angle lens can be a fisheye lens with more than 180° FOVand having an equidistance projection scheme, but it is by no meanslimited to such a fisheye lens. Rather, it can be any rotationallysymmetric wide-angle lens including a catadioptric fisheye lens.Hereinafter, for the sake of notational simplicity, a wide-angle lens isreferred to as a fisheye lens. Said camera body contains photoelectronicsensors such as CCD or CMOS sensors, and it can acquire either a stillimage or a movie. By the said fisheye lens(1012), a real image of theobject plane(1031) is formed on the focal plane(1032). In order toobtain a sharp image, the image sensor plane(1013) must coincide withthe focal plane(1032).

The real image of the objects on the object plane(1031) formed by thefisheye lens(1012) is converted by the image sensor(1013) intoelectrical signals, and displayed as an uncorrected image plane(1034) onthe image display means(1015). This uncorrected image plane(1034)contains a barrel distortion by the fisheye lens. This distorted imageplane can be rectified by the image processing means(1016), and thendisplayed as a processed image plane(1035) on an image displaymeans(1017) such as a computer monitor or a CCTV monitor. Said imageprocessing can be software image processing by a computer, or hardwareimage processing by Field Programmable Gate Arrays (FPGA) or ARM coreprocessors.

An arbitrary rotationally symmetric lens including a fisheye lens doesnot provide said panoramic image or a distortion-free rectilinear image.Therefore, image processing stage is essential in order to obtain adesirable image. FIG. 11 is a conceptual drawing of an uncorrected imageplane(1134) prior to the image processing stage, which corresponds tothe real image on the image sensor plane(1013).

If the lateral dimension of the image sensor plane(1013) is B and thelongitudinal dimension is V, then the lateral dimension of theuncorrected image plane is gB and the longitudinal dimension is gV,where g is proportionality constant.

Uncorrected image plane(1134) can be considered as the image displayedon the image display means without rectification of distortion, and is amagnified image of the real image on the image sensor plane by amagnification ratio g. For example, the image sensor plane of a ⅓-inchCCD sensor has a rectangular shape having a lateral dimension of 4.8 mm,and a longitudinal dimension of 3.6 mm. On the other hand, if themonitor is 48 cm in width and 36 cm in height, then the magnificationratio g is 100. More desirably, the side dimension of a pixel in adigital image is considered as 1. A VGA-grade ⅓-inch CCD sensor haspixels in an array form with 640 columns and 480 rows. Therefore, eachpixel has a right rectangular shape with both width and height measuringas 4.8 mm/640=7.5 μm, and in this case, the magnification ratio g isgiven by 1 pixel/7.5 μm=133.3 pixel/mm. In recapitulation, theuncorrected image plane(1134) is a distorted digital image obtained byconverting the real image formed on the image sensor plane intoelectrical signals.

The first intersection point O on the image sensor plane is theintersection point between the optical axis and the image sensor plane.Therefore, a ray entered along the optical axis forms an image point onthe said first intersection point O. By definition, the point O′ on theuncorrected image plane corresponding to the first intersection point Oin the image sensor plane—hereinafter referred to as the secondintersection point—corresponds to the image point by an incident rayentered along the optical axis.

A second rectangular coordinate systems is assumed wherein x′-axis istaken as the axis that passes through the second intersection point O′on the uncorrected image plane and is parallel to the lateral side ofthe uncorrected image plane, and y′-axis is taken as the axis thatpasses through the said second intersection point O′ and is parallel tothe longitudinal side of the uncorrected image plane. The positivedirection of the x′-axis runs from the left to the right, and thepositive direction of the y′-axis runs from the top to the bottom. Then,the lateral coordinate x′ of an arbitrary point on the uncorrected imageplane(1134) has a minimum value x′₁=gx₁ and a maximum value x′₂=gx₂(i.e., gx₁≦x′≦gx₂). In the same manner, the longitudinal coordinate y′of the said point has a minimum value y′₁=gy₁ and a maximum valuey′₂=gy₂ (i.e., gy₁≦y′≦gy₂).

FIG. 12 is a conceptual drawing of a rectified screen of the currentinvention, wherein the distortion has been removed. In other words, itis a conceptual drawing of a processed image plane(1235) that can bedisplayed on the image display means. The processed image plane(1235)has a rectangular shape, of which the lateral side measuring as W andthe longitudinal side measuring as H. Furthermore, a third rectangularcoordinate system is assumed wherein x″-axis is parallel to the lateralside of the processed image plane, and y″-axis is parallel to thelongitudinal side of the processed image plane. The z″-axis of the thirdrectangular coordinate system coincides with the z-axis of the firstrectangular coordinate system and the z′-axis of the second rectangularcoordinate system. The intersection point O″ between the said z″-axisand the processed image plane—hereinafter referred to as the thirdintersection point—can take an arbitrary position, and it can even belocated outside the processed image plane. Here, the positive directionof the x″-axis runs from the left to the right, and the positivedirection of the y″-axis runs from the top to the bottom.

The first and the second intersection points correspond to the locationof the optical axis. On the other hand, the third intersection pointcorresponds not to the location of the optical axis but to the principaldirection of vision. The principal direction of vision may coincide withthe optical axis, but it is not needed to. Principal direction of visionis the direction of the optical axis of an imaginary panoramic orrectilinear camera corresponding to the desired panoramic or rectilinearimages. Hereinafter, for the sake of notational simplicity, theprincipal direction of vision is referred to as the optical axisdirection.

The lateral coordinate x″ of a third point P″ on the processed imageplane(1235) has a minimum value x″₁ and a maximum value x″₂ (i.e.,x″₁≦x″≦x″₂). By definition, the difference between the maximum lateralcoordinate and the minimum lateral coordinate is the lateral dimensionof the processed image plane(i.e., x″₂−x″₁=W). In the same manner, thelongitudinal coordinate y″ of the third point P″ has a minimum value y″₁and a maximum value y″₂ (i.e., y″₁≦y″≦y″₂). By definition, thedifference between the maximum longitudinal coordinate and the minimumlongitudinal coordinate is the longitudinal dimension of the processedimage plane(i.e., y″₂−y″₁=H).

The following table 1 summarizes corresponding variables in the objectplane, the image sensor plane, the uncorrected image plane, and theprocessed image plane.

TABLE 1 image sensor uncorrected processed surface object plane planeimage plane image plane lateral dimension of B gB W the planelongitudinal V gV H dimension of the plane coordinate system world thefirst the second the third coordinate rectangular rectangularrectangular system coordinate coordinate coordinate system system systemlocation of the nodal point nodal point of nodal point of nodal point ofcoordinate origin of the lens the lens the lens the lens symbol of theorigin O O′ O″ coordinate axes (X, Y, Z) (x, y, z) (x′, y′, z′) (x″, y″,z″) name of the object object point the first point the second point thethird point point or the image points symbol of the object Q P P′ P″point or the image point two dimensional (x, y) (x′, y′) (x″, y″)coordinate of the object point or the image point

On the other hand, if we assume the coordinate of an image point P″ onthe processed image plane(1235) corresponding to an object point with acoordinate (X, Y, Z) in the world coordinate system is (x″, y″), thenthe said image processing means process the image so that the imagepoint corresponding to an incident ray originating from the said objectpoint appears on the said screen with the coordinate (x″, y″), whereinthe lateral coordinate x″ of the image point is given by Eq. 14.

Math Figure 14

x″=cψ  [Math.14]

Here, c is proportionality constant.

Furthermore, the longitudinal coordinate y″ of the said image point isgiven by Eq. 15.

Math Figure 15

y″=cF(μ)  [Math.15]

Here, F(μ) is a monotonically increasing function passing through theorigin. In mathematical terminology, it means that Eqs. 16 and 17 aresatisfied.

Math Figure 16

F(0)=0  [Math.16]

Math Figure 17

$\begin{matrix}{\frac{\partial{F(\mu)}}{\partial\mu} > 0.} & \left\lbrack {{Math}.\mspace{14mu} 17} \right\rbrack\end{matrix}$

The above function F can take an arbitrary form, but the most desirableforms are given by Eqs. 18 through 21.

Math Figure 18

F(μ)=tan μ  [Math.18]

Math Figure 19

$\begin{matrix}{{F(\mu)} = \frac{\tan \; \mu}{\cos \; \chi}} & \left\lbrack {{Math}.\mspace{14mu} 19} \right\rbrack\end{matrix}$Math Figure 20

F(μ)=μ  [Math.20]

Math Figure 21

$\begin{matrix}{{F(\mu)} = {\ln \left\{ {\tan \left( {\frac{\mu}{2} + \frac{\pi}{4}} \right)} \right\}}} & \left\lbrack {{Math}.\mspace{14mu} 21} \right\rbrack\end{matrix}$

FIG. 13 is a schematic diagram of an ordinary car rear viewcamera(1310). For a car rear view camera, it is rather common that awide-angle lens with more than 150° FOV is used, and the optical axis ofthe lens is typically inclined toward the ground plane(1317) asillustrated in FIG. 13. By installing the camera in this way, parkinglane can be easily recognized when backing up the car. Furthermore,since the lens surface is oriented downward toward the ground,precipitation of dust is prevented, and partial protection is providedfrom rain and snow.

To obtain a panoramic image with a horizontal FOV around 180°, it isdesirable to install the image acquisition means(1310) on top of thetrunk of a passenger car, and to align the optical axis at a certainangle with the ground plane. Furthermore, a fisheye lens with more than180° FOV and having an equidistance projection scheme is mostpreferable, and the image display means is desirably installed next tothe driver seat.

It is possible to obtain said panoramic image using a wide-angle camerawith its optical axis inclined toward the ground plane. In this case,the world coordinate system takes the nodal point N of the imagingsystem(1310) as the origin, and takes a vertical line that isperpendicular to the ground plane as the Y-axis, and the Z-axis is setparallel to the car(1351) axle. According to the convention of righthanded coordinate system, the positive direction of the X-axis is thedirection directly plunging into the paper in FIG. 13. Therefore, if thelens optical axis is inclined below the horizon with an angle α, then acoordinate system fixed to the camera has been rotated around the X-axisof the world coordinate system by angle α. This coordinate system isreferred to as the first world coordinate system, and the three axes ofthis first world coordinate system are named as X′, Y′ and Z′-axis,respectively. In FIG. 13, it appears that the first world coordinatesystem has been rotated around the X-axis clockwise by angle α relativeto the world coordinate system. However, considering the direction ofthe positive X-axis, it has been in fact rotated counterclockwise byangle α. Since direction of rotation considers counterclockwise rotationas the positive direction, the first world coordinate system in FIG. 13has been rotated by +α around the X-axis of the world coordinate system.

Regarding the rotation of coordinate system, it is convenient to use theEuler matrices. For this, the coordinate of an object point Q in threedimensional space is designated as a three dimensional vector as givenbelow.

Math Figure 22

$\begin{matrix}{\overset{\_}{Q} = {\begin{pmatrix}X \\Y \\Z\end{pmatrix}.}} & \left\lbrack {{Math}.\mspace{14mu} 22} \right\rbrack\end{matrix}$

Here, {right arrow over (Q)} is the three dimensional vector in theworld coordinate system starting at the origin and ending at the pointQ. Then, the coordinate of a new point obtainable by rotating the pointQ in the space by an angle of −α around the X-axis is given bymultiplying the matrix given in Eq. 23 on the above vector.

Math Figure 23

$\begin{matrix}{{M_{X}(\alpha)} = \begin{pmatrix}1 & 0 & 0 \\0 & {\cos \; \alpha} & {\sin \; \alpha} \\0 & {{- \sin}\; \alpha} & {\cos \; \alpha}\end{pmatrix}} & \left\lbrack {{Math}.\mspace{14mu} 23} \right\rbrack\end{matrix}$

Likewise, the matrix given in Eq. 24 can be used to find the coordinateof a new point which is obtainable by rotating the point Q by angle −βaround the Y-axis, and the matrix given in Eq. 25 can be used to findthe coordinate of a new point which is obtainable by rotating the pointQ by angle −γ around the Z-axis.

Math Figure 24

$\begin{matrix}{{M_{Y}(\beta)} = \begin{pmatrix}{\cos \; \beta} & 0 & {{- \sin}\; \beta} \\0 & 1 & 0 \\{\sin \; \beta} & 0 & {\cos \; \beta}\end{pmatrix}} & \left\lbrack {{Math}.\mspace{14mu} 24} \right\rbrack\end{matrix}$Math Figure 25

$\begin{matrix}{{M_{Z}(\gamma)} = \begin{pmatrix}{\cos \; \gamma} & {\sin \; \gamma} & 0 \\{{- \sin}\; \gamma} & {\cos \; \gamma} & 0 \\0 & 0 & 1\end{pmatrix}} & \left\lbrack {{Math}.\mspace{14mu} 25} \right\rbrack\end{matrix}$

Matrices in Eqs. 23 through 25 can describe the case where thecoordinate system is fixed and the point in space has been rotated, butalso the same matrices can describe the case where the point in space isfixed and the coordinate system has been rotated in the reversedirection. These two cases are mathematically equivalent. Therefore, thecoordinate of a point Q in the first world coordinate system which isobtained by rotating the world coordinate system by angle α around theX-axis as indicated in FIG. 13 is given by Eq. 26.

Math Figure 26

$\begin{matrix}{{\overset{\_}{Q}}^{\prime} = {\begin{pmatrix}X^{\prime} \\Y^{\prime} \\Z^{\prime}\end{pmatrix} = {{M_{X}(\alpha)}\begin{pmatrix}X \\Y \\Z\end{pmatrix}}}} & \left\lbrack {{Math}.\mspace{14mu} 26} \right\rbrack\end{matrix}$

Using the matrix given in Eq. 23, the coordinate in the first worldcoordinate system can be given as follows in terms of the coordinate inthe world coordinate system.

Math Figure 27

X′=X  [Math.27]

Math Figure 28

Y′=Y cos α+Z sin α  [Math.28]

Math Figure 29

Z′=−Y sin α+Z cos α  [Math.29]

Referring to FIG. 13, let's assume that an imaging system has beeninstalled with its optical axis inclined toward the ground plane, andnevertheless, it is desired to obtain a panoramic image that is parallelto the ground plane. If we assume that FIG. 5 is a wide-angle image withα=0°, then FIG. 14 is a wide-angle image with α=30°. In this case, thefollowing algorithm can be used to obtain a panoramic image that isparallel to the ground plane. First, under the assumption that the saidimaging system is parallel to the ground plane, the size (W, H) of theprocessed image plane, the location of the third intersection point O″,and the horizontal field of view Δψ are determined. For simplicity ofargument, the horizontal FOV is assumed as symmetric. Then, thehorizontal incidence angle ψ corresponding to the lateral coordinate x″on the processed image plane and the vertical incidence angle δcorresponding to the longitudinal coordinate y″ are given by Eqs. 30through 32.

Math Figure 30

$\begin{matrix}{A = \frac{W}{\Delta\psi}} & \left\lbrack {{Math}.\mspace{14mu} 30} \right\rbrack\end{matrix}$Math Figure 31

$\begin{matrix}{\psi = \frac{x^{''}}{A}} & \left\lbrack {{Math}.\mspace{14mu} 31} \right\rbrack\end{matrix}$Math Figure 32

$\begin{matrix}{\delta = {F^{- 1}\left( \frac{y^{''}}{A} \right)}} & \left\lbrack {{Math}.\mspace{14mu} 32} \right\rbrack\end{matrix}$

Here, the angle χ is 0°. Therefore, the function F(δ) is identical tothe function F(μ) given in Eqs. 16 through 21. In other words, it istrue that F(δ)=F(μ). Next, it is assumed that an incident ray havingthese horizontal and vertical incidence angles has been originated froman object point on a hemisphere with a radius S having its center at thenodal point of the lens. Then, the coordinate of the said object pointin the world coordinate system is given by Eqs. 33 through 36.

Math Figure 33

X=S cos δ sin ψ  [Math.33]

Math Figure 34

Y=S sin δ  [Math.34]

Math Figure 35

Z=S cos δ cos ψ  [Math.35]

Math Figure 36

S=√{square root over (X ² +Y ² +Z ²)}

The coordinate of this object point in the first world coordinate systemis given by Eqs. 27 through 29. The X′, Y′ and Z′-axes of this firstworld coordinate system are parallel to the x, y, and z-axes of thefirst rectangular coordinate system, respectively. Therefore, the zenithand the azimuth angles of the incident ray are given by Eqs. 37 and 38.

Math Figure 37

$\begin{matrix}{\theta = {\cos^{- 1}\left( \frac{Z^{\prime}}{S} \right)}} & \left\lbrack {{Math}.\mspace{14mu} 37} \right\rbrack\end{matrix}$Math Figure 38

$\begin{matrix}{\varphi = {\tan^{- 1}\left( \frac{Y^{\prime}}{X^{\prime}} \right)}} & \left\lbrack {{Math}.\mspace{14mu} 38} \right\rbrack\end{matrix}$

Finally, two dimensional rectangular coordinate (x′, y′) of the secondpoint in the uncorrected image plane having these zenith and azimuthangles can be obtained from the two dimensional polar coordinate by Eqs.39 and 40.

Math Figure 39

x′=gr(θ)cos φ  [Math.39]

Math Figure 40

y′=gr(θ)sin φ  [Math.40]

Using Eqs. 9 through 40, a panoramic image having an ideal projectionscheme can be extracted from an image acquired using a fisheye lensexhibiting the distortion aberration. First, depending on the user'sneed, a desirable size (W, H) of the panoramic image, and the locationof the third intersection point O″ are determined. The said thirdintersection point can be located even outside the processed imageplane. In other words, the range of the lateral coordinate (x″₁≦x″≦x″₂)on the processed image plane as well as the range of the longitudinalcoordinate (y″₁≦y″<y″₂) can take arbitrary real numbers. Also, thehorizontal FOV Δψ of this panoramic image (i.e., the processed imageplane) is determined. The functional form of F(δ) dictating thedesirable projection scheme along the vertical direction is determined,as well. Then, the horizontal incidence angle w and the verticalincidence angle δ of an incident ray corresponding to the third point onthe panoramic image having a rectangular coordinate (x″, y″) can beobtained using Eqs. 30 through 32. Then, the zenith angle θ and theazimuth angle φ of an incident ray having the said horizontal incidenceangle and the vertical incidence angle are calculated using Eqs. 37 and38. Next, the real image height r corresponding to the zenith angle θ ofthe incident ray is obtained using Eq. 9. Utilizing the real imageheight r, the magnification ratio g, and the azimuth angle φ of theincident ray, the rectangular coordinate (x′, y′) of the image point onthe uncorrected image plane is obtained using Eqs. 39 and 40. In thisprocedure, the coordinate of the second intersection point on theuncorrected image plane, or equivalently the first intersection point onthe image sensor plane, has to be accurately determined. Such a locationof the intersection point can be easily found using various methodsincluding image processing method. Since such technique is well known tothe people in this field, it will not be described in this document.Finally, the video signal (i.e., RGB signal) by the fisheye lens fromthe image point having said rectangular coordinate is given as the videosignal for the image point on the panoramic image having the rectangularcoordinate (x″, y″). A panoramic image having an ideal projection schemecan be obtained by image processing for all the image points on theprocessed image plane by the above-described method.

Considering the fact that all the image sensors and display devices aredigital devices, image processing procedure must use the following setof equations. First, a desirable size of the processed image plane andthe location (I_(o), J_(o)) of the third intersection point aredetermined. Here, the location (I_(o), J_(o)) of the third intersectionpoint refers to the pixel coordinate of the third intersection point O″.Conventionally, the coordinate of a pixel on the upper left corner of adigitized image is defined as (1, 1) or (0, 0). In the currentinvention, we will assume that the coordinate of the pixel on the upperleft corner is given as (1, 1). Then, the horizontal incidence angle ψ₁corresponding to the lateral pixel coordinate of J=1 and the horizontalincidence angle ψ_(Jmax) corresponding to the lateral pixel coordinateof J=J_(max) are determined. Then, for every pixel (I, J) on theprocessed image plane, the horizontal incidence angle ψ_(J) and thevertical incidence angle δ₁ are calculated using Eqs. 41 and 42.

Math Figure 41

$\begin{matrix}{\psi_{J} = {\frac{\psi_{Jmax} - \psi_{1}}{J_{\max} - 1}\left( {J - J_{o}} \right)}} & \left\lbrack {{Math}.\mspace{14mu} 41} \right\rbrack\end{matrix}$Math Figure 42

$\begin{matrix}{\delta_{I} = {F^{- 1}\left\{ {\frac{\psi_{Jmax} - \psi_{1}}{J_{\max} - 1}\left( {I - I_{o}} \right)} \right\}}} & \left\lbrack {{Math}.\mspace{14mu} 42} \right\rbrack\end{matrix}$

From the above horizontal and vertical incidence angles, the coordinateof an imaginary object point in the world coordinate system iscalculated using Eqs. 43 through 45.

Math Figure 43

X _(I,J)=cos δ_(I) sin ψ_(J)  [Math.43]

Math Figure 44

Y _(I,J)=sin δ_(I)  [Math.44]

Math Figure 45

Z _(I,J)=cos δ_(I) cos ψ_(J)  [Math.45]

Here, the object distance S does not affect the final outcome and thusit is assumed as 1 for the sake of simplicity. From this coordinate ofthe object point in the world coordinate system, the coordinate of theobject point in the first world coordinate system is calculated fromEqs. 46 through 48.

Math Figure 46

X′ _(I,J) =X _(I,J)  [Math.46]

Math Figure 47

Y′ _(I,J) =Y _(I,J) cos α+Z _(I,J) sin α  [Math.47]

Math Figure 48

Z′ _(I,J) =−Y _(I,J) sin α+Z _(I,J) cos α  [Math.48]

From this coordinate, the zenith angle θ_(I,J) and the azimuth angleΦ_(I,J) of the incident ray are calculated using Eqs. 49 and 50,

Math Figure 49

θ_(I,J)=cos⁻¹(Z′ _(I,J))  [Math.49]

Math Figure 50

$\begin{matrix}{\varphi_{I,J} = {\tan^{- 1}\left( \frac{Y_{I,J}^{\prime}}{X_{I,J}^{\prime}} \right)}} & \left\lbrack {{Math}.\mspace{14mu} 50} \right\rbrack\end{matrix}$

Next, the image height r_(I,J) on the image sensor plane is calculatedusing Eq. 51.

Math Figure 51

r _(I,J) =r(θ_(I,J))  [Math.51]

Then, the position (K_(o), L_(o)) of the second intersection point onthe uncorrected image plane and the magnification ratio g are used tofind the position of the second point on the uncorrected image plane.

Math Figure 52

x′ _(I,J) =L _(o) +gr _(I,J) cos(φ_(I,J))  [Math.52]

Math Figure 53

y′ _(I,J) =K _(o) +gr _(I,J) sin(φ_(I,J))  [Math.53]

Once the position of the corresponding second point has been found,diverse interpolation methods such as the nearest-neighbor method, thebilinear interpolation method, and the bicubic interpolation method canbe used to obtain a panoramic image.

FIG. 15 is a panoramic image obtained using this method, and acylindrical projection scheme has been employed. As can be seen fromFIG. 15, a perfect panoramic image has been obtained despite the factthat the optical axis is not parallel to the ground plane. Using such apanoramic imaging system as a car rear view camera, the backside of avehicle can be entirely monitored without any dead spot.

One point which needs special attention when using such an imagingsystem as a car rear view camera is the fact that for a device (i.e., acar) of which the moving direction is the exact opposite of the opticalaxis direction of the image acquisition means, it can cause a greatconfusion to the driver if a panoramic image obtained by above describedmethod is displayed without any further processing. Since the car rearview camera is heading toward the backside of the car, the right end ofthe car appears as the left end on the monitor showing the imagescaptured by the rear view camera. However, the driver can fool himselfby thinking that the image is showing the left end of the car from hisown viewpoint of looking at the front end of the car, and thus, there isa great danger of possible accidents. To prevent such a perilousconfusion, it is important to switch the left and the right sides of theimage obtained using a car rear view camera before displaying it on themonitor. The video signal S′(I, J) for the pixel in a mirrored (i.e.,the left and the right sides are exchanged) processed image plane with acoordinate (I, J) is given by the video signal S(I, J_(max)−J+1) fromthe pixel in the processed image plane with coordinate (I, J_(max)−J+1).

Math Figure 54

S′(I,J)=S(I,J _(max) −J+1)  [Math.54]

Identical image acquisition means can be installed near the room mirror,frontal bumper, or the radiator grill in order to be used as a recordingcamera connected to a car black box for the purpose of recordingvehicle's driving history.

Above embodiment has been described in relation to a car rear viewcamera, but it must be obvious that the usefulness of the inventiondescribed in this embodiment is not limited to a car rear view camera.

Said invention in reference 17 provides image processing algorithms forextracting mathematically accurate panoramic images and devicesimplementing the algorithms. In many cases, however, distortion-freerectilinear image can be of more value. Or, it can be more satisfactorywhen panoramic images and rectilinear images are both available.

-   [reference 1] J. F. Blinn and M. E. Newell, “Texture and reflection    in computer generated images”, Communications of the ACM, 19,    542-547 (1976).-   [reference 2] N. Greene, “Environment mapping and other applications    of world projections”, IEEE Computer Graphics and Applications, 6,    21-29 (1986).-   [reference 3] S. D. Zimmermann, “Omniview motionless camera    orientation system”, U.S. Pat. No. 5,185,667, date of patent Feb. 9,    1993.-   [reference 4] E. Gullichsen and S. Wyshynski, “Wide-angle image    dewarping method and apparatus”, U.S. Pat. No. 6,005,611, date of    patent Dec. 21, 1999.-   [reference 5] E. W. Weisstein, “Cylindrical Projection”,    http://mathworld.wolfram.com/CylindricalProjection.html.-   [reference 6] W. D. G. Cox, “An introduction to the theory of    perspective—part 1”, The British Journal of Photography, 4, 628-634    (1969).-   [reference 7] G. Kweon, K. Kim, Y. Choi, G. Kim, and S. Yang,    “Catadioptric panoramic lens with a rectilinear projection scheme”,    Journal of the Korean Physical Society, 48, 554-563 (2006).-   [reference 8] G. Kweon, Y. Choi, G. Kim, and S. Yang, “Extraction of    perspectively normal images from video sequences obtained using a    catadioptric panoramic lens with the rectilinear projection scheme”,    Technical Proceedings of the 10th World Multi-Conference on    Systemics, Cybernetics, and Informatics, 67-75 (Orlando, Fla., USA,    June, 2006).-   [reference 9] H. L. Martin and D. P. Kuban, “System for    omnidirectional image viewing at a remote location without the    transmission of control signals to select viewing parameters”, U.S.    Pat. No. 5,384,588, date of patent Jan. 24, 1995.-   [reference 10] F. Oxaal, “Method and apparatus for performing    perspective transformation on visible stimuli”, U.S. Pat. No.    5,684,937, date of patent Nov. 4, 1997.-   [reference 11] F. Oxaal, “Method for generating and interactively    viewing spherical image data”, U.S. Pat. No. 6,271,853, date of    patent Aug. 7, 2001.-   [reference 12] N. L. Max, “Computer graphics distortion for IMAX and    OMNIMAX projection”, Proc. NICOGRAPH, 137-159 (1983).-   [reference 13] P. D. Bourke, “Synthetic stereoscopic panoramic    images”, Lecture Notes in Computer Graphics (LNCS), Springer, 4270,    147-155 (2006).-   [reference 14] G. Kweon and M. Laikin, “Fisheye lens”, Korean patent    10-0888922, date of patent Mar. 10, 2009.-   [reference 15] G. Kweon, Y. Choi, and M. Laikin, “Fisheye lens for    image processing applications”, J. of the Optical Society of Korea,    12, 79-87 (2008).-   [reference 16] G. Kweon and M. Laikin, “Wide-angle lenses”, Korean    patent 10-0826571, date of patent Apr. 24, 2008.-   [reference 17] G. Kweon, “Methods of obtaining panoramic images    using rotationally symmetric wide-angle lenses and devices thereof”,    Korean patent 10-0882011, date of patent Jan. 29, 2009.-   [reference 18] W. K. Pratt, Digital Image Processing, 3rd edition    (John Wiley, New York, 2001), Chap. 19.

DISCLOSURE OF INVENTION Technical Problem

The purpose of the present invention is to provide image processingalgorithms for extracting natural looking panoramic images andrectilinear images from digitized images acquired using a cameraequipped with a wide-angle lens which is rotationally symmetric about anoptical axis and devices implementing such algorithms.

Technical Solution

The present invention provides image processing algorithms that areaccurate in principle based on the geometrical optics principleregarding image formation by wide-angle lenses with distortion andmathematical definitions of panoramic and rectilinear images.

Advantageous Effects

By applying accurate image processing algorithms on images obtainedusing a rotationally symmetric wide-angle lens, panoramic images thatappear most natural to the naked eye as well as rectilinear images canbe obtained.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a conceptual drawing of the latitude and the longitude.

FIG. 2 is a conceptual drawing of a map with an equi-rectangularprojection scheme.

FIG. 3 is a conceptual drawing illustrating a cylindrical projectionscheme.

FIG. 4 is a conceptual drawing illustrating the real projection schemeof a general rotationally symmetric lens.

FIG. 5 is an exemplary image produced by a computer assuming that afisheye lens with an equidistance projection scheme has been used totake the picture of an imaginary scene.

FIG. 6 is a diagram showing the optical structure of a refractivefisheye lens with a stereographic projection scheme along with thetraces of rays.

FIG. 7 is a diagram showing the optical structure of a catadioptricfisheye lens with a stereographic projection scheme along with thetraces of rays.

FIG. 8 is a diagram showing the optical structure of a catadioptricpanoramic lens with a rectilinear projection scheme along with thetraces of rays.

FIG. 9 is a conceptual drawing of the world coordinate system of theinvention of a prior art.

FIG. 10 is a schematic diagram of a panoramic imaging system of theinvention of a prior art.

FIG. 11 is a conceptual drawing of an uncorrected image plane.

FIG. 12 is a conceptual drawing of a processed image plane that can beshown on an image display means.

FIG. 13 is a schematic diagram of a car rear view camera employing apanoramic imaging system of the invention of a prior art.

FIG. 14 is an exemplary fisheye image captured by an inclined imagingsystem.

FIG. 15 is an exemplary panoramic image extracted from FIG. 14.

FIG. 16 is a conceptual drawing illustrating the rectilinear projectionscheme of the first embodiment of the present invention.

FIG. 17 is a conceptual drawing illustrating the change in field of viewas the relative position of the processed image plane is changed.

FIG. 18 is the conceptual drawing of a processed image plane accordingto the first embodiment of the present invention.

FIG. 19 is an exemplary rectilinear image with a horizontal FOV of 120°extracted from FIG. 5.

FIG. 20 is an exemplary rectilinear image after the application of slideand zoom operations.

FIG. 21 is an exemplary image of an interior scene captured using afisheye lens of the invention of a prior art.

FIG. 22 is a panoramic image with a horizontal FOV of 190° and followinga cylindrical projection scheme extracted from the fisheye image givenin FIG. 21.

FIG. 23 is a rectilinear image with a horizontal FOV of 60° extractedfrom the fisheye image given in FIG. 21.

FIG. 24 is a schematic diagram of a panoramic camera phone embodying theconception of the present invention.

FIG. 25 is an exemplary rectilinear image obtained after applyingpan•tilt operation to the fisheye image given in FIG. 5.

FIG. 26 is an exemplary rectilinear image obtained after applyingpan•tilt operation to the fisheye image given in FIG. 21.

FIG. 27 is another exemplary image of an interior scene captured using afisheye lens of the invention of a prior art.

FIG. 28 is an exemplary rectilinear image obtained after applyingpan•tilt operation to the fisheye image given in FIG. 27.

FIG. 29 is an exemplary rectilinear image obtained after applyingtilt•pan operation to the fisheye image given in FIG. 27.

FIG. 30 is a conceptual drawing illustrating the most general secondworld coordinate system in rectilinear projection schemes.

FIG. 31 is a schematic diagram showing the relation between the secondworld coordinate system and the processed image plane in rectilinearprojection schemes.

FIG. 32 is a schematic diagram of a car rear view camera employingslide•pan•tilt operations.

FIG. 33 is an exemplary rectilinear image obtained after applying tiltoperation to the fisheye image given in FIG. 5.

FIG. 34 is a conceptual drawing of an imaging system with large pan•tiltangles in absence of slide operation.

FIG. 35 is a conceptual drawing of an imaging system with large pan•tiltangles with a proper mix of slide operation.

FIG. 36 is an exemplary rectilinear image obtained after applying slideand tilt operations to the fisheye image given in FIG. 5.

FIG. 37 is an exemplary image of an outdoor scene captured using afisheye lens of the invention of a prior art.

FIG. 38 is a panoramic image following a Mercator projection schemeextracted from the fisheye image given in FIG. 37.

FIG. 39 is an exemplary rectilinear image obtained after applying tiltoperation to the fisheye image given in FIG. 37.

FIG. 40 is an exemplary rectilinear image obtained after applyingpan•tilt operations to the fisheye image given in FIG. 37.

FIG. 41 is a schematic diagram of an imaging system for a vehicleembodying the conception of the present invention.

FIG. 42 is a schematic diagram of an imaging system for monitoring thesurroundings of a building embodying the conception of the presentinvention.

FIG. 43 is another exemplary image of an outdoor scene captured using afisheye lens of the invention of a prior art.

FIG. 44 is an exemplary rectilinear image extracted from the fisheyeimage given in FIG. 43.

FIG. 45 is a schematic diagram for understanding the rectilinear imagegiven in FIG. 44.

FIG. 46 is a schematic diagram illustrating a desirable method ofinstalling cameras on a vehicle.

FIG. 47 is another exemplary image of an outdoor scene captured using afisheye lens of the invention of a prior art.

FIG. 48 is an exemplary rectilinear image extracted from the fisheyeimage given in FIG. 47.

FIG. 49 is another schematic diagram illustrating a desirable method ofinstalling cameras on a vehicle.

FIG. 50 is another schematic diagram of an imaging system for monitoringthe surroundings of a building embodying the conception of the presentinvention.

FIG. 51 is a conceptual drawing of the object plane in a multipleviewpoint panoramic imaging system.

FIG. 52 is an exemplary multiple viewpoint panoramic image extractedfrom the fisheye image given in FIG. 5.

FIG. 53 is a conceptual drawing illustrating the notion of multipleviewpoint panoramic image.

FIG. 54 is another exemplary multiple viewpoint panoramic imageextracted from the fisheye image given in FIG. 5.

FIG. 55 is an exemplary multiple viewpoint panoramic image extractedfrom the fisheye image given in FIG. 37.

FIG. 56 is an exemplary multiple viewpoint panoramic image extractedfrom the fisheye image given in FIG. 27.

FIG. 57 is another exemplary image of an interior scene captured using afisheye lens of the invention of a prior art.

FIG. 58 is an exemplary panoramic image extracted from the fisheye imagegiven in FIG. 57.

FIG. 59 is an exemplary multiple viewpoint panoramic image extractedfrom the fisheye image given in FIG. 57.

FIG. 60 is a schematic diagram of a desirable embodiment of the imageprocessing means of the present invention.

MODE FOR THE INVENTION

Referring to FIG. 16 through FIG. 60, the preferable embodiments of thepresent invention will be described in detail.

First Embodiment

FIG. 16 is a conceptual drawing illustrating the rectilinear projectionscheme of the first embodiment of the present invention. By definition,a lens with a rectilinear projection scheme is a distortion-free lens,and in mathematical viewpoint, the characteristics of a rectilinear lensare considered identical to those of a pinhole camera. To acquire suchan image with a rectilinear projection scheme, we assume an objectplane(1631) and a processed image plane(1635) in the world coordinatesystem as shown in FIG. 16.

The imaging system of the present embodiment is heading in an arbitrarydirection, and the third rectangular coordinate system takes the opticalaxis(1601) of the imaging system as the negative z″-axis, and the nodalpoint of the lens as the origin. Image sensor plane has a rectangularshape with a lateral width B and a longitudinal height V, and the imagesensor plane is a plane perpendicular to the optical axis. On the otherhand, the processed image plane has a rectangular shape with a lateralwidth W and a longitudinal height H. The x-axis of the first rectangularcoordinate system, the x′-axis of the second rectangular coordinatesystem, the x″-axis of the third rectangular coordinate system and theX-axis of the world coordinate system are all parallel to the lateralside of the image sensor plane. Furthermore, the z-axis of the firstrectangular coordinate system, the z′-axis of the second rectangularcoordinate system, and the z″-axis of the third rectangular coordinatesystems are all identical to each other and opposite to the Z-axis ofthe world coordinate system.

In the present embodiment, the processed image plane is assumed to belocated at a distance s″ from the nodal point of the lens. In arectilinear projection scheme, the shape of the object plane(1631) isalso a plane perpendicular to the optical axis, and the image of objectson the object plane is faithfully reproduced on the processed imageplane(1635) with both the lateral and the longitudinal scales preserved.The ideal projection scheme of a rectilinear lens is identical to theprojection scheme of a pinhole camera. Considering the simplegeometrical characteristics of a pinhole camera, it is convenient toassume that the shape and the size of the object plane(1631) areidentical to those of the processed image plane. Therefore, the distancefrom the object plane(1631) to the nodal point N of the lens is alsoassumed as s″.

FIG. 17 is a conceptual drawing illustrating how the horizontal field ofview Δψ of a lens is changed by the relative position of the processedimage plane(1735). In FIG. 17( a), the position of the processed imageplane is symmetric with respect to the optical axis. Therefore, theimage displayed on the processed image plane has a symmetricalhorizontal field of view. On the other hand, in FIG. 17( b), theprocessed image plane(1735) has been laterally displaced with respect tothe optical axis, and the FOVs are different for the left and the rightsides. Such an operation is useful when it is desired to change themonitored area without changing the principal direction of vision.Physically, it corresponds to laterally displacing the image sensor withrespect to the optical axis. In the present invention, such an operationwill be referred to as a slide operation.

On the other hand, FIG. 17( c) shows the case where the distance s″between the nodal point and the processed image plane has beenincreased. In this case, as illustrated in the figure, the field of viewbecomes narrower, and only a small region is monitored. Physically, thiscorresponds to a zoom operation. Therefore, by changing the relativeposition of the processed image plane with respect to the optical axis,and the distance to the nodal point, slide and zoom effects can beachieved.

FIG. 18 illustrates the case where the intersection point O between theimage sensor plane and the optical axis, or equivalently the thirdintersection point O “on the processed image plane corresponding to thefirst intersection point O does not coincide with the center C” of theprocessed image plane. Therefore, it corresponds to an imaging systemwith slide operation. In two-dimensional rectangular coordinate systemhaving the third intersection point as the origin, the coordinate of thesaid center C″ is given as (x″_(c), y″_(c)). Since the lateral dimensionof the processed image plane is W, the lateral coordinate with respectto the center C″ has a minimum value x″₁=−W/2 and a maximum valuex″₂=W/2. Considering the coordinate of the center C″ on top of this, therange of the lateral coordinate of the processed image plane has aminimum value x″₁=x″_(c)−W/2 and a maximum value x″₂=x″_(c)+W/2.Likewise, the range of the longitudinal coordinate has a minimum valuey″₁=y″_(c)−H/2 and a maximum value y″₂=y″_(c)+H/2.

The distance between the third intersection point O″ on the processedimage plane to the third point P″, or the image height r″ is given byEq. 55.

Math Figure 55

r″=√{square root over ((x″)²+(y″)²)}{square root over ((x″)²+(y″)²)}

Since the virtual distance between the nodal point of the lens to theprocessed image plane is s″, an incident ray arriving at the third pointby the rectilinear lens has a zenith angle given by Eq. 56.

Math Figure 56

$\begin{matrix}{\theta = {\tan^{- 1}\left( \frac{r^{''}}{s^{''}} \right)}} & \left\lbrack {{Math}.\mspace{14mu} 56} \right\rbrack\end{matrix}$

On the other hand, the azimuth angle of the said incident ray is givenby Eq. 57.

Math Figure 57

$\begin{matrix}{\varphi = {\varphi^{''} = {\tan^{- 1}\left( \frac{y^{''}}{x^{''}} \right)}}} & \left\lbrack {{Math}.\mspace{14mu} 57} \right\rbrack\end{matrix}$

Therefore, when an incident ray having the said zenith angle and thesaid azimuth angle forms an image point on the image sensor plane by theimage forming properties of the lens, the coordinate of the image pointis given by Eqs. 58 and 59.

Math Figure 58

x′=gr(θ)cos φ  [Math.58]

Math Figure 59

y′=gr(θ)sin φ  [Math.59]

Therefore, it is only necessary to substitute the signal value of thethird point on the processed image plane by the signal value of theimage point on the uncorrected image plane having such a rectangularcoordinate.

In absence of a slide action, in other words, when the position of thethird intersection point coincides with the center of the processedimage plane, the following equation given in Eq. 60 is satisfied.

Math Figure 60

$\begin{matrix}{s^{''} = {\frac{\frac{W}{2}}{\tan \left( \frac{\Delta\psi}{2} \right)} = \frac{\frac{H}{2}}{\tan \left( \frac{\Delta\delta}{2} \right)}}} & \left\lbrack {{Math}.\mspace{14mu} 60} \right\rbrack\end{matrix}$

Using Eq. 60, the virtual distance s″ of the processed image planehaving given horizontal or vertical FOVs can be calculated. Purely forthe purpose of convenience, the size (W, H) of the processed image planeand the horizontal FOV is first determined. Then, the virtual distanceof the processed image plane is obtained, and from this distance, thesymmetrical vertical FOV Δδ is automatically determined. On the otherhand, if the coordinate of the center of the processed image plane isgiven by (x″_(c), y″_(c)) as a result of a slide operation, then thefollowing Eqs. 61 and 62 are satisfied.

Math Figure 61

$\begin{matrix}\begin{matrix}{s^{''} = \frac{x^{''}}{\tan \; \psi}} \\{= \frac{x_{c}^{''} - \frac{W}{2}}{\tan \; \psi_{\min}}} \\{= \frac{x_{c}^{''} + \frac{W}{2}}{\tan \; \psi_{\max}}} \\{= \frac{W}{{\tan \; \psi_{\max}} - {\tan \; \psi_{\min}}}}\end{matrix} & \left\lbrack {{Math}.\mspace{14mu} 61} \right\rbrack\end{matrix}$Math Figure 62

$\begin{matrix}\begin{matrix}{s^{''} = \frac{y^{''}}{\tan \; \delta}} \\{= \frac{y_{c}^{''} - \frac{H}{2}}{\tan \; \delta_{\min}}} \\{= \frac{y_{c}^{''} + \frac{H}{2}}{\tan \; \delta_{\max}}} \\{= \frac{H}{{\tan \; \delta_{\max}} - {\tan \; \delta_{\min}}}}\end{matrix} & \left\lbrack {{Math}.\mspace{14mu} 62} \right\rbrack\end{matrix}$

Here, ψ_(max) and ψ_(min) are the maximum and the minimum incidenceangles in the horizontal direction, and likewise, δ_(max) and δ_(min)are the maximum and the minimum incidence angles in the verticaldirection. Furthermore, irrespective of the position of the center, thefollowing relation given in Eq. 63 must be satisfied all the time.

Math Figure 63

$\begin{matrix}{\frac{W}{{\tan \; \psi_{\max}} - {\tan \; \psi_{\min}}} = \frac{H}{{\tan \; \delta_{\max}} - {\tan \; \delta_{\min}}}} & \left\lbrack {{Math}.\mspace{14mu} 63} \right\rbrack\end{matrix}$

Similar to the said embodiment of the invention of a prior art,considering the facts that all the image sensors and the display devicesare digital devices, it is convenient to use the following equations inimage processing procedure. First of all, the size (I, J_(max)) of theprocessed image plane and the horizontal FOV Δψ prior to any slideoperation are determined. Then, the pixel distance s″ between the nodalpoint of the lens and the processed image plane can be obtained usingEq. 64.

Math Figure 64

$\begin{matrix}{s^{''} = \frac{J_{\max} - 1}{2{\tan \left( \frac{\Delta\psi}{2} \right)}}} & \left\lbrack {{Math}.\mspace{14mu} 64} \right\rbrack\end{matrix}$

Furthermore, the coordinate of the center of the processed image planeis given by Eq. 65.

Math Figure 65

$\begin{matrix}{\left( {I_{o},J_{o}} \right) = \left( {\frac{1 + I_{\max}}{2},\frac{1 + J_{\max}}{2}} \right)} & \left\lbrack {{Math}.\mspace{20mu} 65} \right\rbrack\end{matrix}$

Here, Eq. (65) reflects the convention that the coordinate of the pixelon the upper left corner of a digital image is given as (1, 1).

Next, according to the needs, the displacement (ΔI, ΔJ) of the saidcenter from the third intersection point is determined. Once suchpreparatory stage has been finished, the zenith angle given in Eq. 66and the azimuth angle given in Eq. 67 are calculated for every pixel onthe processed image plane.

Math Figure 66

$\begin{matrix}{\theta_{I,J} = {\tan^{- 1}\left\{ \frac{\sqrt{\left( {I - I_{o} + {\Delta \; I}} \right)^{2} + \left( {J - J_{o} + {\Delta \; J}} \right)^{2}}}{s^{''}} \right\}}} & \left\lbrack {{Math}.\mspace{14mu} 66} \right\rbrack\end{matrix}$Math Figure 67

$\begin{matrix}{\varphi_{I,J} = {\tan^{- 1}\left( \frac{I - I_{o} + {\Delta \; I}}{J - J_{o} + {\Delta \; J}} \right)}} & \left\lbrack {{Math}.\mspace{14mu} 67} \right\rbrack\end{matrix}$

Next, the image height r_(u) on the image sensor plane is calculatedusing Eq. 68.

Math Figure 68

r _(I,J) =r(θ_(I,J))  [Math.68]

Next, the position of the second point on the uncorrected image plane iscalculated using the position of the second intersection point on theuncorrected image plane and the magnification ratio g.

Math Figure 69

x′ _(I,J) =L _(o) +gr _(I,J) cos(φ_(I,J))  [Math.69]

Math Figure 70

y′ _(I,J) =K _(o) +gr _(I,J) sin(φ_(I,J))  [Math.70]

Once the position of the corresponding second point has been found, therectilinear image can be obtained using the previously describedinterpolation methods.

FIG. 19 is a rectilinear image extracted from the fisheye image given inFIG. 5, wherein the lateral dimension of the processed image plane is240 pixels, the longitudinal dimension is 180 pixels, the horizontal FOVis 120°, and there is no slide operation. As can be seen from FIG. 19,all the straight lines are captured as straight lines. On the otherhand, FIG. 20 shows a rectilinear image, of which the parameters areidentical to those of FIG. 19 except for the fact that the center of theprocessed image plane has been slid 70 pixels along the lateraldirection and −30 pixels along the longitudinal direction.

On the other hand, FIG. 21 is an exemplary image of an interior scene,which has been acquired by aligning the optical axis of a fisheye lenswith 190° FOV described in references 14 and 15 parallel to the groundplane. The real projection scheme of this fisheye lens is described indetail in the said references. On the other hand, FIG. 22 is a panoramicimage having a cylindrical projection scheme extracted from the fisheyeimage given in FIG. 21. Here, the width:height ratio of the processedimage plane is 16:9, the position of the third intersection pointcoincides with the center of the processed image plane, and thehorizontal FOV is set as 190°. As can be seen from FIG. 22, all thevertical lines are captured as vertical lines and all the objects appearnatural. Slight errors are due to the error in aligning the optical axisparallel to the ground plane, and the error in experimentallydetermining the position of the optical axis on the uncorrected imageplane.

On the other hand, FIG. 23 is a rectilinear image extracted from FIG. 21with the width:height ratio of 4:3, wherein the position of the thirdintersection point coincides With the center of the processed imageplane, and the horizontal FOV is 60°. Here, it can be seen that all thestraight lines in the world coordinate system are captured as straightlines in the processed image plane.

Second Embodiment

FIG. 24 shows one exemplary application wherein such an invention can beused. The built-in camera module is a very important factor indetermining the market value of a cellular phone. If a panoramic picturecan be taken with a cellular phone(2414), then the customer satisfactionwill be greatly increased. Ordinary lens currently mounted in cameraphones is comprised of 2˜4 pieces of double aspherical lens elements,and has a mega pixel grade resolution with a typical FOV of 60°. On theother hand, it is possible to realize a fisheye lens with a FOV between120° and 180° using 3˜5 pieces of double aspherical lens elements.Furthermore, since high end processor such as an ARM core processor istypically embedded in cellular phones, it can handle the imageprocessing load required to embody the conception of the currentinvention. Therefore, a panoramic camera phone can be realized byinstalling a wide-angle lens with more than 120° FOV, and endowing theimage processing function to the electronic circuitry of the cellularphone. The resolution of the image sensor for such application ispreferably more than 1M pixels. By utilizing such a camera phone, theuser can obtain a panoramic image such as given in FIG. 22 or arectilinear image such as given in FIG. 23.

A complication arises if it is desired to endow more than 180° FOV tothe built-in lens for a cellular phone. If the lens does not protrudefrom the wall of the cellular phone, then the wall occlude the view ofthe lens even if the lens itself has a FOV which is larger than 180°.Therefore, in order to obtain a FOV which is greater than 180°, the lensmust protrude from the wall of the cellular phone. However, due to thecharacteristics of a cellular phone being carried by the user all thetime, a protruding lens can be scratched or stained. Considering thisfact, a lens cover can be procured to cover the lens when the built-incamera is not in use. Another method is to make the camera module tocome out from the wall of the cellular phone only when the phone camerais to be used. Finally, the easiest method is to make the FOV of thelens less than 170°. For such a FOV, the lens needs not protrude fromthe wall, and a fixed lens module is sufficient. Considering the factthat the horizontal FOV of a panoramic camera with a rotating lens isonly 120°, it is apparent that such a FOV can be enough for the customersatisfaction.

Identical techniques can be used in PC camera or web camera, and thecomputing power for the necessary image processing can be provided by aPC connected to the PC camera, or a computer of another user connectedto the Internet, or a network server. Furthermore, a fisheye image canbe acquired using a digital camera equipped with a fisheye lens, then apanoramic image or a rectilinear image can be extracted using an imageediting software running on PC.

Third Embodiment

If a camera equipped with a fisheye lens with 180° FOV is installed onan interior wall, then practically, there is no dead zone of securitymonitoring. It is because the region not captured by the camera is thewall that needs not be security-monitored. However, as has been statedpreviously, an image by a fisheye lens causes psychological discomfortdue to its barrel distortion. On the other hand, although an ultrawide-angle image extracted in the manner of the first embodiment of thepresent invention captures most of the interior scene, captured objectsfar from the optical axis do not appear natural to the naked eye. Inthis case, the most natural looking image is a rectilinear image thatcan be acquired by heading the camera to the direction of the object.

A camera that can physically provide such an image is a camera equippedwith a rectilinear lens and mounted on a pan•tilt stage. Since cameracan be oriented to the direction that needs most attention, a mostsatisfactory image can be provided. Furthermore, images can becontinuously produced while dynamically tracking a moving object such asa cat or an intruder. A method of realizing such functionality withsoftware is provided as follows.

Due to the mathematical properties of three-dimensional rotations,different images are obtained depending on which of the two operations,namely a pan or a tilt operation has been taken first. In thisembodiment, we assume that a pan operation has been taken first.Furthermore, as in the first embodiment, slide and zoom operations areallowed. However, slide and zoom operations must precede the panoperation, and the tilt operation is assumed to follow the panoperation. The coordinate system describing the objects prior to the panoperation is the world coordinate system, the coordinate system afterthe pan operation is the first world coordinate system, and thecoordinate system after the tilt operation is the second worldcoordinate system. The X″, Y″ and Z″-axes of the second world coordinatesystem are respectively parallel to the x, y and z-axes of the firstrectangular coordinate system.

First of all, as in the first embodiment of the present invention, thesize (W, H) of the processed image plane and the horizontal FOV Δψ priorto the slide operation are determined. Then, the distance s″ of theprocessed image plane is determined by Eq. 60. Next, a proper amount(x″_(c), y″_(c)) of slide operation is determined in order to obtain adesirable horizontal FOV (ψ_(min)≦ψ≦ψ_(max)) and a vertical FOV(δ_(min)≦δ≦δ_(max)). Every object point on the object planecorresponding to the image point on the processed image plane has azenith angle given by Eq. 56 and an azimuth angle given by Eq. 57. Then,the coordinate of an object point Q at an object distance R from thenodal point of the lens is given by Eqs. 71 through 74.

Math Figure 71

X=R sin θ cos φ  [Math.71]

Math Figure 72

Y=R sin θ sin φ  [Math.72]

Math Figure 73

Z=R cos θ  [Math.73]

Math Figure 74

R=√{square root over (X ² +Y ² +Z ²)}

Here, the object distance can take an arbitrary value, and can be takenas 1 for the sake of simplicity.

In the coordinate system of the current invention, pan operation is arotational operation around the Y-axis, and tilt operation is arotational operation around the X′-axis. The coordinate of a new pointwhich corresponds to an object point in the world coordinate systemhaving a coordinate (X, Y, Z) rotated around the Y-axis by angle −β isgiven as (X′, Y′, Z′), and the coordinate of yet another point whichcorresponds to the said new point rotated around the X′-axis by angle −αis given as (X″, Y″, Z″). Utilizing the Euler matrices given in thefirst embodiment of the present invention, the coordinate of the newpoint is given by Eq. 75.

Math Figure 75

$\begin{matrix}{{\overset{\rightarrow}{Q}}^{''} = {\begin{pmatrix}X^{''} \\Y^{''} \\Z^{''}\end{pmatrix} = {{M_{X}(\alpha)}{M_{Y}(\beta)}{\begin{pmatrix}X \\Y \\Z\end{pmatrix}.}}}} & \left\lbrack {{Math}.\mspace{14mu} 75} \right\rbrack\end{matrix}$

The following relation holds due to the mathematical properties of Eulermatrices.

Math Figure 76

M _(X′)(α)=M _(Y)(β)M _(X)(α)M _(Y)(−β)  [Math.76]

Therefore, Eq. 75 can be expressed in a simpler form as given below.

Math Figure 77

$\begin{matrix}{\begin{pmatrix}X^{''} \\Y^{''} \\Z^{''}\end{pmatrix} = {{M_{Y}(\beta)}{M_{X}(\alpha)}\begin{pmatrix}X \\Y \\Z\end{pmatrix}}} & \left\lbrack {{Math}.\mspace{14mu} 77} \right\rbrack\end{matrix}$

By calculating these rotational matrices, the coordinate of the newpoint is given by Eqs. 78 through 80.

Math Figure 78

X″=X cos β+Y sin β sin α+Z sin β cos α  [Math.78]

Math Figure 79

Y″=Y cos α−Z sin α  [Math.79]

Math Figure 80

Z″=−X sin β+Y cos β sin α+Z cos β cos α  [Math.80]

The procedure for finding the coordinate of the first point from thecoordinate of this new point is identical to that of the firstembodiment.

Considering the fact that all the image sensors and display means aredigital devices, image processing for a rectilinear projection schemecan be done using the procedure outlined below.

First, the size (I_(max), J_(max)) of the processed image plane, thehorizontal field of view Δψ, the vertical field of view Δδ, and thepixel distance s″ from the nodal point of the lens to the processedimage plane satisfying Eq. 81 are obtained.

Math Figure 81

$\begin{matrix}{s^{''} = {\frac{J_{\max} - 1}{2{\tan \left( \frac{\Delta\psi}{2} \right)}} = \frac{I_{\max} - 1}{2{\tan \left( \frac{\Delta\delta}{2} \right)}}}} & \left\lbrack {{Math}.\mspace{14mu} 81} \right\rbrack\end{matrix}$

The coordinate of the center of the processed image plane is given byEq. 82.

Math Figure 82

$\begin{matrix}{\left( {I_{o},J_{o}} \right) = \left( {\frac{1 + I_{\max}}{2},\frac{1 + J_{\max}}{2}} \right)} & \left\lbrack {{Math}.\mspace{20mu} 82} \right\rbrack\end{matrix}$

Next, according to the needs, the displacement (LM, M) of the saidcenter from the third intersection point is determined. Once suchpreparatory stage has been completed, the zenith angle given in Eq. 83and the azimuth angle given in Eq. 84 are calculated for all the pixelson the processed image plane.

Math Figure 83

$\begin{matrix}{\theta_{I,J} = {\tan^{- 1}\left\{ \frac{\sqrt{\left( {I - I_{o} + {\Delta \; I}} \right)^{2} + \left( {J - J_{o} + {\Delta \; J}} \right)^{2}}}{s^{''}} \right\}}} & \left\lbrack {{Math}.\mspace{14mu} 83} \right\rbrack\end{matrix}$Math Figure 84

$\begin{matrix}{\varphi_{I,J} = {\tan^{- 1}\left( \frac{I - I_{o} + {\Delta \; I}}{J - J_{o} + {\Delta \; J}} \right)}} & \left\lbrack {{Math}.\mspace{14mu} 84} \right\rbrack\end{matrix}$

Next, the coordinate of the object point Q on the object plane havingthese zenith and azimuth angles is calculated using Eqs. 85 through 87.

Math Figure 85

X _(I,J)=sin θ_(I,J) cos φ_(I,J)  [Math.85]

Math Figure 86

Y _(I,J)=sin θ_(I,J) sin φ_(I,J)  [Math.86]

Math Figure 87

Z _(I,J)=cos θ_(I,J)  [Math.87]

Here, the object distance has been assumed as 1. Next, the coordinate ofthis object point in the second world coordinate system is calculatedusing Eqs. 88 through 90.

Math Figure 88

X″ _(I,J) =X _(I,J) cos β+Y _(I,J) sin β sin α+Z _(I,J) sin β cosα  [Math.88]

Math Figure 89

Y″ _(I,J) =Y _(I,J) cos α−Z _(I,J) sin α  [Math.89]

Math Figure 90

Z″ _(I,J) =−X _(I,J) sin β+Y _(I,J) cos β sin α+Z _(I,J) cos β cosα  [Math.90]

From this coordinate, the zenith angle θ_(I,J) and the azimuth angleφ_(I,J) of the incident ray are calculated using Eqs. 91 and 92.

Math Figure 91

θ_(I,J)=cos⁻¹(Z″ _(I,J))  [Math.91]

Math Figure 92

$\begin{matrix}{\varphi_{I,J} = {\tan^{- 1}\left( \frac{Y_{I,J}^{''}}{X_{I,J}^{''}} \right)}} & \left\lbrack {{Math}.\mspace{14mu} 92} \right\rbrack\end{matrix}$

Next, the image height r_(u) on the image sensor plane is calculatedusing Eq. 93.

Math Figure 93

r _(I,J) =r(θ_(I,J))  [Math.93]

Next, the position of the second point on the uncorrected image plane isobtained from the position (K_(o), L_(O)) of the second intersectionpoint on the uncorrected image plane and the magnification ratio g.

Math Figure 94

x′ _(I,J) =L _(o) +gr _(I,J) cos(φ_(I,J))  [Math.94]

Math Figure 95

y′ _(I,J) =K _(o) +gr _(I,J) sin(φ_(I,J))  [Math.95]

Once the position of the second point has been found, the rectilinearimage can be obtained using interpolation methods identical to thosedescribed in the first embodiment.

FIG. 25 is a rectilinear image extracted from the fisheye image given inFIG. 5, wherein the lateral dimension of the processed image plane is240 pixels, the longitudinal dimension is 180 pixels, the horizontal FOVprior to the slide•pan•tilt operations is 120°, there is no slideoperation, and the rotation angles are given as α=β=30°. As can be seenfrom FIG. 25, all the straight lines are captured as straight lines. Onthe other hand, FIG. 26 is a rectilinear image extracted from thefisheye image given in FIG. 21, wherein the lateral dimension of theprocessed image plane is 1280 pixels, the longitudinal dimension is 960pixels, the horizontal FOV prior to the slide•pan•tilt operations is70°, there is no slide operation, and the rotation angles are given asα=40° and β=20°.

Fourth Embodiment

FIG. 27 is another exemplary image of an interior scene captured using afisheye lens described in references 14 and 15, wherein the optical axisof the fisheye lens has been inclined downward from the horizontal planetoward the floor (i.e., nadir) by 45°. On the other hand, FIG. 28 is arectilinear image extracted from FIG. 27 with pan•tilt operations.Specifically, the lateral dimension of the processed image plane is 1280pixels, the longitudinal dimension is 960 pixels, the horizontal FOVprior to the slide•pan•tilt operations is 60°, there is no slideoperation, and the rotation angles are given as α=45° and β=50°. Fromthe figure, however, it can be seen that vertically extended objectssuch as the bookshelves appear slanted. In such a case, a satisfactoryimage can be obtained when the tilt operation precedes the panoperation. In other words, the method of extracting a rectilinear imagepresented in the third embodiment is a method wherein slide operation istaken first, and then the pan and the tilt operations follow insequence. However, depending on the application area, it can be moreadvantageous if the order of the pan and the tilt operations areexchanged. Similar to the third embodiment of the present invention, thecoordinate of a new point obtainable by taking the tilt and the panoperations in sequence is given by Eq. 96.

Math Figure 96

$\begin{matrix}{\begin{pmatrix}X^{''} \\Y^{''} \\Z^{''}\end{pmatrix} = {{{M_{Y}(\beta)}{M_{X}(\alpha)}\begin{pmatrix}X \\Y \\Z\end{pmatrix}} = {{M_{X}(\alpha)}{M_{Y}(\beta)}\begin{pmatrix}X \\Y \\Z\end{pmatrix}}}} & \left\lbrack {{Math}.\mspace{14mu} 96} \right\rbrack\end{matrix}$

By evaluating these rotational matrices, the coordinate of the objectpoint in the second world coordinate system is given by Eqs. 97 through99.

Math Figure 97

X″=X cos β+Z sin β  [Math.97]

Math Figure 98

Y″=X sin α sin β+Y cos α−Z sin α cos β  [Math.98]

Math Figure 99

Z″=−X cos α sin β+Y sin α+Z cos α cos β  [Math.99]

The procedure for finding the coordinate of the second point from thecoordinate of the object point is identical to that given in the thirdembodiment.

Considering the fact that all the image sensors and display means aredigital devices, image processing for the rectilinear projection schemecan be done using the procedure outlined below.

First, the size (I_(max), J_(max)) of the processed image plane, thehorizontal field of view Δψ, the vertical field of view Δδ, and thepixel distance s″ from the nodal point of the lens to the processedimage plane satisfying Eq. 100 are obtained.

Math Figure 100

$\begin{matrix}{s^{''} = {\frac{J_{\max} - 1}{2\; {\tan\left( \frac{\Delta\psi}{2} \right)}} = \frac{I_{\max} - 1}{2\; {\tan\left( \frac{\Delta\delta}{2} \right)}}}} & \left\lbrack {{Math}.\mspace{14mu} 100} \right\rbrack\end{matrix}$

The coordinate of the center of the processed image plane is given byEq. 101.

Math Figure 101

$\begin{matrix}{\left( {I_{o},J_{o}} \right) = \left( {\frac{1 + I_{\max}}{2},\frac{1 + J_{\max}}{2}} \right)} & \left\lbrack {{Math}.\mspace{14mu} 101} \right\rbrack\end{matrix}$

Next, according to the needs, the displacement (ΔI, ΔJ) of the saidcenter from the third intersection point is determined. Once suchpreparatory stage has been completed, the zenith angle given in Eq. 102and the azimuth angle given in Eq. 103 are calculated for all the pixelson the processed image plane.

Math Figure 102

$\begin{matrix}{\theta_{I,J} = {\tan^{- 1}\left\{ \frac{\sqrt{\left( {I - I_{o} + {\Delta \; I}} \right)^{2} + \left( {J - J_{o} + {\Delta \; J}} \right)^{2}}}{s^{''}} \right\}}} & \left\lbrack {{Math}.\mspace{14mu} 102} \right\rbrack\end{matrix}$Math Figure 103

$\begin{matrix}{\varphi_{I,J} = {\tan^{- 1}\left( \frac{I - I_{o} + {\Delta \; I}}{J - J_{o} + {\Delta \; J}} \right)}} & \left\lbrack {{Math}.\mspace{14mu} 103} \right\rbrack\end{matrix}$

Next, the coordinate of the object point Q on the object plane havingthese zenith and azimuth angles are calculated using Eqs. 104 through106.

Math Figure 104

X _(I,J)=sin θ_(I,J) cos φ_(I,J)  [Math.104]

Math Figure 105

Y _(I,J)=sin θ_(I,J) sin φ_(I,J)  [Math.105]

Math Figure 106

Z _(I,J)=cos θ_(I,J)  [Math.106]

Next, the coordinate of this object point in the second world coordinatesystem is calculated using Eqs. 107 through 109.

Math Figure 107

X″ _(I,J) =X _(I,J) cos β+Z _(I,J) sin β  [Math.107]

Math Figure 108

Y″ _(I,J) =X _(I,J) sin α sin β+Y _(I,J) cos α−Z _(I,J) sin α cosβ  [Math.108]

Math Figure 109

Z″ _(I,J) =−X _(I,J) cos α sin β+Y _(I,J) sin α+Z _(I,J) cos α cosβ  [Math.109]

From this coordinate, the zenith angle θ_(I,J) and the azimuth angleφ_(I,J) of the incident ray are calculated using Eqs. 110 and 111.

Math Figure 110

θ_(I,J)=cos⁻¹(Z″ _(I,J))  [Math.110]

Math Figure 111

$\begin{matrix}{\varphi_{I,J} = {\tan^{- 1}\left( \frac{Y_{I,J}^{''}}{X_{I,J}^{''}} \right)}} & \left\lbrack {{Math}.\mspace{14mu} 111} \right\rbrack\end{matrix}$

Next, the image height r_(u) on the image sensor plane is calculatedusing Eq. 112.

Math Figure 112

r _(I,J) =r(θ_(I,J))  [Math.112]

Next, the position of the second point on the uncorrected image plane isobtained from the position (K_(o), L_(o)) of the second intersectionpoint on the uncorrected image plane and the magnification ratio g.

Math Figure 113

x′ _(I,J) =L _(o) +gr _(I,J) cos(φ_(I,J))  [Math.113]

Math Figure 114

y′ _(I,J) =K _(o) +gr _(I,J) sin(φ_(I,J))  [Math.114]

Once the position of the corresponding second point has been found, therectilinear image can be obtained using interpolation methods identicalto those described in the first and the second embodiments.

FIG. 29 is a rectilinear image extracted from the fisheye image given inFIG. 27 with tilt and pan operations. Specifically, the lateraldimension of the processed image plane is 1280 pixels, the longitudinaldimension is 960 pixels, the horizontal FOV prior to the slide•pan•tiltoperations is 60°, there is no slide operation, and the rotation anglesare given as α=45° and α=50°. It can be seen from the figure that thebookshelves appear to stand in an upright position.

Such rectilinear projection schemes can be summarized as follows. Theworld coordinate system employed by the imaging system of the currentinvention is schematically shown in FIG. 9. FIG. 30 shows the secondworld coordinate system (X″, Y″, Z″), which is obtainable by rotatingand translating the world coordinate system (X, Y, Z) shown in FIG. 9.Specifically, the origin N″ of the second world coordinate system shownin FIG. 30 is the origin N in the world coordinate system translated byΔX along the X-axis direction, and by ΔY along the Y-axis direction.Then, the X″-axis and the Y″-axis are the X-axis and the Y-axis rotatedaround the Z-axis by angle γ, respectively, after the abovetranslational operations. FIG. 31 illustrates that the coordinate (X″,Y″) of the object point in the second world coordinate systems isproportional to the coordinate (x″, y″) of the image point on theprocessed image plane. In other words, a relation given in Eq. 115 holdstrue.

Math Figure 115

$\begin{matrix}{\frac{X^{''}}{x^{''}} = \frac{Y^{''}}{y^{''}}} & \left\lbrack {{Math}.\mspace{14mu} 115} \right\rbrack\end{matrix}$

Generally speaking, a rectilinear projection scheme is a scheme whereina second world coordinate system (X″, Y″, Z″) exist such that therelation given in Eq. 115 holds true for all the image points (x″, y″)on the processed image plane, where the second world coordinate system(X″, Y″, Z″) is a coordinate system obtainable by rotating andtranslating the said world coordinate system (X, Y, Z) in thethree-dimensional space by arbitrary number of times and in randomorders.

Fifth Embodiment

FIG. 32 is a schematic diagram of a car rear view camera utilizing thewide-angle imaging system of the current invention. The panoramicimaging system of the invention of a prior art described in reference 17can be used as a car rear view camera in order to completely eliminatethe dead zones in monitoring. Said fisheye lens can be installed insidea passenger car(3251) trunk for the purpose of monitoring the backsideof a car without a dead zone. It can also be installed at the bumpers orat the rear window. However, considering the purpose of a rear viewcamera, the top of the trunk will be the ideal place to install a rearview camera. Furthermore, for buses and trucks, the rear view camera isdesirably installed at the top of the rear end of the vehicle.

On the other hand, at the time of parking or backing up the car, theimage of the immediate back area of the car will be most helpful ratherthan the view of a remote scenery. To acquire an image of the far behindarea while driving the car, the optical axis(3201) of the imageacquisition means(3210) will be aligned parallel to the groundplane(3217). Therefore, in order to visually check the obstacles lyingbehind the car or the parking lane, the pan•tilt operations presented inthe first through the third embodiments of the present invention will behelpful.

FIG. 33 is a rectilinear image extracted from the fisheye image given inFIG. 5. The lateral dimension of the processed image plane is 240pixels, the longitudinal dimension is 180 pixels, the horizontal FOVprior to the slide•pan•tilt operations is 120°, there is no slideoperation, and the rotation angles are given as α=90° and β=0°.Therefore, it corresponds to a case where a tilt operation has beentaken without a pan operation. By the way, as can be seen from FIG. 33,only half of the screen contains meaningful image. The reason can beunderstood referring to FIG. 34. FIG. 34 has been drawn under theassumption that the FOV of the wide-angle lens of the image acquisitionmeans is 180°. In absence of slide•pan•tilt operations as illustrated inFIG. 34( a), the processed image plane(3435) following a rectilinearprojection scheme as well as the object plane(3431) are perpendicular tothe optical axis, and all the object points on the object plane arewithin the FOV of the said lens. When the tilt angle becomes 90° as inFIG. 34( b), then half of the processed image plane(3435) and half ofthe object plane(3431) lie outside the FOV of the lens. Therefore, whena tilt operation with a tilt angle of 90° is taken purely in software,then a region where there is no visual information exists occupy half ofthe screen. Furthermore, referring to FIG. 32 and FIG. 34, the regionoutside the FOV of the lens is the vehicle's body, and therefore it isnot really meaningful to monitor that area.

On the other hand, FIG. 35 shows a way to resolve such problem. If it isdesired to make the pan or the tilt angle equal to 90° in order to makethe principal direction of vision perpendicular to the optical axis,then as schematically shown in FIG. 35( a), it is preferable to apply aslide operation to the processed image plane(3535) first, so that objectplane(3531) lies at the opposite side of the direction with respect tothe optical axis along which a rotational operation is intended. Then,even if the tilt angle becomes 90° as in FIG. 35( b), the object planeremains within the FOV of the lens and a satisfactory rectilinear imageis obtained. FIG. 36 is an example of a wide-angle image obtained byapplying the slide operation before the tilt operation.

FIG. 37 is yet another fisheye image obtained using the said fisheyelens with the optical axis aligned parallel to the ground plane. Acamera equipped with the said fisheye lens has been set-up near thesidewall of a large bus and headed outward from the bus. The height ofthe camera was comparable to the height of the bus. As can be seen fromFIG. 37, all the objects within a hemisphere including the sidewall ofthe bus have been captured in the image.

FIG. 38 is a panoramic image having a horizontal FOV of 190° extractedfrom FIG. 37. The used projection scheme is the Mercator projectionscheme, and the coordinate of the third intersection point is (I_(o),J_(o))=(1, J_(max)/2). On the other hand, FIG. 39 is a rectilinear imageextracted from FIG. 37, of which the horizontal FOV is 90°, the tiltangle is −90°, and the size of the slide operation is (ΔI,ΔJ)=(−I_(max)/2.0). On the other hand, FIG. 40 is another rectilinearimage extracted from FIG. 37, of which the horizontal FOV is 90°, andthe size of the slide operation is (ΔI, ΔI)=(0, J_(max)/2). To obtain animage showing the front side of the bus, first, −100° pan operation hasbeen applied and then subsequently −30° tilt operation has been applied.

From the FIGS. 38 through 40, it can be seen that such an imaging systemis useful as a camera for vehicles. FIG. 41 shows a schematic diagram ofa car using such an imaging system. At least one camera(4110L, 4110R,4110B) is installed at the outside wall of a vehicle(4151) such as apassenger car or a bus, and preferably identical fisheye lenses areinstalled on all the cameras. Furthermore, this camera can be installedparallel to the ground plane, vertical to the ground plane, or at anangle to the ground plane. Which one among these options is takendepends on the particular application area.

The fisheye images acquired from these cameras are gathered by the imageprocessing means(4116), and displayed on the image display means(4117)after proper image processing operation has been taken, and it can besimultaneously recorded on an image recording means(4118). This imagerecording means can be a DVR (Digital Video Recorder).

Another device connected to this image processing means(4116) is asituation awareness module or an image selection means. This imageselection means receives signal from the electronic circuitry of the carwhich indicates whether the car is running forward, backing up, or thedriver gave the change-lane signal to turn to the left or to the right.Even when the vehicle is moving forward, it may be necessary todifferentiate between a fast and a slow driving. Independent of allthese signals, the driver can manually override the signals bymanipulating the menus on the dashboard in order to force the projectionscheme and the parameters of the image displayed on the image displaymeans. Based upon such signals supplied by the image selection means,the image processing means provide the most appropriate images suitablefor the current driving situation such as the images from the left-sidecamera(4110L) or the rear view camera(4110B).

Even if the rear view camera(4110B) is the only camera installed at thevehicle, the situation awareness module is still useful. For example,when the car is moving forward, panoramic image such as the one given inFIG. 22 or a rectilinear image parallel to the ground plane such as theone given in FIG. 23 can be extracted from the fisheye image acquired bythe rear view camera and shown to the driver. When right turn signallamp is lit, a pan•tilt image such as the one given in FIG. 40 can beshown to the driver. And, when back-up signal lamp is lit, then a tiltedimage such as the one given in FIG. 39 can be shown to the driver inorder to avoid possible accidents.

When several cameras are simultaneously used as in FIG. 41, then each ofthe raw images from the respective camera can be separatelyimage-processed, and then the processed images can be seamlessly joinedtogether to form a single panoramic image or a rectilinear image. Inthis case, a more satisfactory image can be shown to the driver such asa bird's eye view providing image of the vehicle and its surroundings asif taken from the air.

In summary, imaging system for vehicles includes an image acquisitionmeans for acquiring an uncorrected image plane which is equipped with awide-angle lens rotationally symmetric about an optical axis, imageprocessing means producing a processed image plane from the uncorrectedimage plane, image selection means determining the projection scheme andthe parameters of the processed image plane, and image display meansdisplaying the processed image plane on a screen with a rectangularshape. The projection schemes of the said processed image plane includea panoramic projection scheme and a rectilinear projection scheme. Avehicle is a device having an imaging system for vehicles.

The world coordinate system for this imaging system takes the nodalpoint of the wide-angle lens as the origin and a vertical line passingthrough the said origin as the Y-axis. When the coordinate of an imagepoint on the said rectangular shaped screen corresponding to an objectpoint in the world coordinate system having a coordinate (X, Y, Z) isgiven as (x″, y″), then, said panoramic projection scheme is a schemewherein straight lines parallel to the said Y-axis in the worldcoordinate system all appear as straight lines parallel to the y″-axison the said screen, and two objects points having identical angulardistance on the X-Z plane in the said world coordinate system has anidentical distance along the x″-axis on the said screen. On the otherhand, said rectilinear projection scheme is a scheme wherein anarbitrary straight line in the said world coordinate system appears as astraight line on the said screen.

Such a principle can be used in many other areas beyond the field ofimaging systems for vehicles. For example, said device is a cameraphone, and said image selection means is a menu button on the cameraphone that a user can select. Or, said device is a PC camera, said imageacquisition means is a USB CMOS camera with a fisheye lens, and saidimage processing means is software running on a computer. Or, saiddevice is a digital camera, said image processing means is softwarerunning on a computer. In this case, said image acquisition means andsaid image processing means are two means which are separated not onlyphysically but also in time. In other words, image acquisition means isa digital camera equipped with a fisheye lens and image processing meansis image editing software running on a computer, Furthermore, imageselection means is a menu button on the image editing software, and saidimage display means is of course the PC monitor.

Sixth Embodiment

FIG. 42 is a schematic diagram of a building monitoring system using theimaging system of the fourth embodiment. As shown in FIG. 42, one camerais installed at a high point on each wall of the building. Four cameraswill be needed in general since typical building has a rectangularshape. If four walls of the building face the east, the west, the north,and the south, respectively, then a rectangular shaped region of theground plane adjacent to the east wall of the building is assigned asthe object plane for the east camera(4210E). Likewise, whilecorresponding object planes(4231W, 4231N, 4231S) are assigned for thewest camera(4210W), the north camera(4210N), and the southcamera(4210S), respectively, it is ensured that adjacent object planesare tightly neighbored to each other so that the building's surroundingscan be monitored in its entirety without a dead spot. Desirably, eachcamera is installed heading vertically downward toward the ground plane,but a similar effect can be obtained by installing the cameras at anangle. After rectilinear images have been extracted from the imagesacquired by the installed cameras, the region of the image correspondingto the object plane illustrated in FIG. 42 are trimmed out. When, theseimages are all put together, a satisfactory image offering the view ofthe building's surroundings in a bird's eye view is obtained. In thiscase, also, situation awareness module(4219) displays various images onthe image display means(4217) corresponding to the operation of thesecurity personnel.

In the present embodiment, the procedure of assembling the rectilinearimages extracted from each direction is called an image registration.Such an image registration technique is described in detail in reference18, and basic principle is to calculate a correlation function from atleast partially overlapping two sub-images and from the peak value ofthe correlation function, the amount of relative translation or rotationbetween the two images necessary for the two images to match at its bestis determined.

Seventh Embodiment

FIG. 43 is an image acquired from a camera equipped with a fisheye lens,which is installed at the top of a passenger vehicle as illustrated inFIG. 13 with the camera tilt angle of −45°. As can be seen from FIG. 43,the FOV is so wide that even the vehicle's number plate can be read. Onthe other hand, FIG. 44 is a rectilinear image extracted from FIG. 43having a horizontal FOV of 120° and −45° tilt angle. Here, it can beseen that the parking lane on the ground plane is clearly visible.However, it can be seen that the car rear bumper appears larger than theparking lane. Such a rectilinear image will not be of a much use for aparking assistance means. The reason the image appears in this way canbe understood by referring to FIG. 45.

FIG. 45 is a schematic diagram of a case where a car rear viewcamera(4510) is installed near the top of a passenger car(4551)'s trunk.The characteristics of a camera providing a rectilinear image isidentical to that of a pinhole camera. In FIG. 45, the width of theparking lane(4555) is needless to say wider than the width of thepassenger car, or the rear bumper(4553). Therefore, the passenger carcan be completed contained within the parking lane. However, when viewedfrom a rectilinear lens installed at the center, the bumper appearswider than the parking lane due to the height of the bumper. Viewed fromthe camera in FIG. 45, the boundary(4563) of the parking lane appearsnecessarily narrower than the boundary of the bumper(4563), and it hasnothing to do with the projection scheme of the lens, but it is theresult of a pure viewpoint. For the parking lane to appear wider thanthe bumper, the heights of the two objects must be similar, but this ispractically impossible.

FIG. 46 is a schematic diagram of an imaging system for vehiclesresolving the aforementioned problems. The cameras for vehicles areinstalled at each corner of the vehicle, and the optical axes arealigned perpendicular to the ground plane. For example, it can beinstalled at the corner of the bumper, which is a protruding end of thecar. In this state of camera installation, precise location(4663) of thevehicle with respect to the ground plane is obtained independent of theheight of the bumper, and therefore the distance to the parking lane canbe accurately estimated while trying to park the vehicle. However, it isnot mandatory to make the optical axis of the image acquisition meansperpendicular to the ground plane, and it can even be parallel to theground plane. Even when the optical axis of the image acquisition meansmakes an angle to the ground plane, an image identical to that of acamera with the optical axis perpendicular to the ground plane can beobtained by a proper combination of pan, tilt and slide operations.

FIG. 47 is an image obtained using a camera equipped with a fisheyelens, which is installed near the top of a recreational vehicle with theoptical axis perpendicular to the ground plane, and FIG. 48 is arectilinear image extracted from such a fisheye image. From FIG. 48, itcan be seen that the vehicle and the parking lane appear well separatedfrom each other.

Such an imaging system will be particularly useful for a vehicle with ahigh stature such as a bus or a truck as schematically illustrated inFIG. 49. Since it is installed at a height higher than the stature of aman, the rectilinear images appear more natural, it is easier tomaintain, and there is a less chance of breakage or defilement.

Such an imaging system for vehicles is characterized as comprised of the1st and the 2nd image acquisition means for acquiring the 1st and the2nd uncorrected image planes using wide-angle lenses rotationallysymmetric about optical axes, an image processing means for extractingthe 1st and the 2nd processed image planes following rectilinearprojection schemes from the said 1st and the 2nd uncorrected imageplanes and subsequently generating a registered processed image planefrom the 1st and the 2nd processed image planes, and an image displaymeans for displaying the said registered processed image plane on arectangular shaped screen. Said device is an automobile having saidimaging systems for vehicles.

Preferably, said 1st and 2nd image acquisition means are installed attwo corners of the rear end of the vehicle heading downward toward theground plane, and said registered processed image plane contains theview of the said two corners.

On the other hand, FIG. 50 shows an imaging system for monitoring theoutside of a building using the same principle. Here, the said device isthe building itself. Such an imaging system for monitoring the outsideof a building is characterized as comprised of the 1st through the 4thimage acquisition means(5010NW, 5010NE, 5010SW, 5010SE) for acquiringthe 1st through the 4th uncorrected image planes using wide-angle lensesrotationally symmetric about optical axes, an image processingmeans(5016) for extracting the 1st through the 4th processed imageplanes following rectilinear projection schemes from the said 1stthrough the 4th uncorrected image planes and subsequently generating aregistered processed image plane from the 1st through the 4th processedimage planes, and an image display means(5017) for displaying the saidregistered processed image plane on a rectangular shaped screen.

Preferably, the said 1st through the 4th image acquisition means areinstalled at the four corners of the building heading downward towardthe ground plane, and the said registered processed image plane containsthe view of the said four corners.

Eighth Embodiment

Panoramic imaging system of the reference 17 requires a directionsensing means in order to constantly provide natural-looking panoramicimages irrespective of the inclination of the device having the imagingsystem with respect to the ground plane. However, it may happen thatadditional installation of a direction sensing means may be difficult interms of cost, weight, or volume for some devices such as motorcycle orunmanned aerial vehicle. FIG. 51 is a conceptual drawing of an objectplane providing a multiple viewpoint panorama that can be advantageouslyused in such cases.

Object plane of the current embodiment has a structure where more thantwo sub object planes are joined together, where each of the sub objectplane is a planar surface by itself. Although FIG. 51 is illustrated asa case where three sub object planes, namely 5131-1, 5131-2 and 5131-3are used, a more general case of using n sub object planes can be easilyunderstood as well. In order to easily understand the currentembodiment, a sphere with a radius T centered on the nodal point of thelens is assumed. If a folding screen is set-up around the sphere whilekeeping the folding screen to touch the sphere, then this folding screencorresponds to the object plane of the current embodiment, Therefore,the n sub object planes are all at the same distance T from the nodalpoint of the lens. As a consequence, all the sub object planes have theidentical zoom ratio or the magnification ratio.

In FIG. 51 using three sub object planes, the principal direction ofvision(5101-1) of the 1st sub object plane(5131-1) makes an angle ofψ₁₋₂ with the principal direction of vision(5101-2) of the 2nd subobject plane(5131-2), and the principal direction of vision(5101-3) ofthe 3rd sub object plane(5131-3) makes an angle of ψ₃₋₄ with theprincipal direction of vision(5101-2) of the 2nd sub objectplane(5131-2). The range of the horizontal FOV of the 1st sub objectplane is from a minimum value ψ₁ to a maximum value ψ₂, and the range ofthe horizontal FOV of the 2nd sub object plane is from a minimum valueψ₂ to a maximum value ψ₃. By having the horizontal FOVs of adjacent subobject planes seamlessly continued, a natural looking multiple viewpointpanoramic image can be obtained.

FIG. 52 is an example of a multiple viewpoint panoramic image extractedfrom FIG. 5, wherein each of the sub processed image plane is 240 pixelswide along the lateral direction and 180 pixels high along thelongitudinal direction. Furthermore, the horizontal FOV of each of thesub object plane or the sub processed image plane is 60°, there is noslide operation for each sub object plane, and the distance to the subobject plane is identical for all the sub object planes. The pan anglefor the 3rd sub object plane on the left side is −60°, the pan angle forthe 2nd sub object plane in the middle is 0°, and the pan angle for the1st sub object plane on the right side is 60°. Since, each of the threesub object planes has a horizontal FOV of 60°, the said three sub objectplanes comprise a multiple viewpoint panoramic image having anhorizontal FOV of 180° as a whole. As can be seen from FIG. 52, all thestraight lines appear as straight lines in each sub object plane.

FIG. 53 is a conceptual drawing illustrating the definition of amultiple viewpoint panoramic image. An imaging system providing amultiple viewpoint panoramic image includes an image acquisition meansfor acquiring an uncorrected image plane which is equipped with awide-angle lens rotationally symmetric about an optical axis, an imageprocessing means for extracting a processed image plane from the saiduncorrected image plane, and an image display means for displaying theprocessed image plane on a screen with a rectangular shape.

Said processed image plane is a multiple viewpoint panoramic image,wherein the said multiple viewpoint panoramic image is comprised of the1st through the nth sub rectilinear image planes laid out horizontallyon the said screen, n is a natural number larger than 2, an arbitrarystraight line in the world coordinate system having the nodal point ofthe wide-angle lens as the origin appears as a straight line(5381A) onany of the 1st through the nth sub rectilinear image plane, and anystraight line in the world coordinate system appearing on more than twoadjacent sub rectilinear image planes appears as a connected linesegments(5381B-1, 53818-2, 5381B-3).

On the other hand, FIG. 54 is another multiple viewpoint panoramic imageextracted from FIG. 5. This is a multiple viewpoint panoramic imageobtained by first tilting the three sub object planes by −55°, and thensubsequently panning the sub object planes by the same condition appliedto FIG. 52. As can be seen from FIG. 54, even if the vertical lines inthe world coordinate system are not parallel to the longitudinal side ofthe image sensor plane, they still appear as straight lines in theprocessed image plane. FIG. 55 is a multiple viewpoint panoramic imageextracted from FIG. 37, and FIG. 56 is a multiple viewpoint panoramicimage extracted from FIG. 27. In FIG. 55 and FIG. 56, each of the threesub object plane has a horizontal FOV of 190°/3.

FIG. 57 is another example of a fisheye image, and it shows the effectof installing a fisheye lens with 190° FOV on the center of the ceilingof an interior. FIG. 58 is a panoramic image extracted from the fisheyeimage given in FIG. 57. On the other hand, FIG. 59 is a multipleviewpoint panoramic image extracted from FIG. 57. Each sub object planehas a horizontal FOV of 360°/4, and it has been obtained by tilting allthe sub object planes by −90° first, and subsequently panning the subobject planes by necessary amounts. From FIG. 59, it can be seen thatsuch an imaging system will be useful as an indoor security camera. Thefollowing is the MatLab code used to obtain the image in FIG. 59.

% Image processing algorithm for generating a multiple viewpointpanoramic image from a fisheye image. % format long % % *********** Realprojection scheme ******************************* coeff = [−3.101406e−2,7.612269e−2, −1.078742e−1, 3.054932e−2, 1.560778, 0.0]; % % *** Read inthe graphic image ********** picture = imread(‘image’, ‘jpg’); [Kmax,Lmax, Qmax] = size(picture); CI = double(picture) + 1; % Lo = 1040; Ko =750; gain = 312.5; % 1/2-inch sensor % % Draw an empty canvas Jmax =round(Lmax / 4); % canvas width Imax = Jmax; % canvas height Jo =(Jmax + 1) / 2; Io = 1; EI = zeros(Imax, 4 * Jmax + 3, 3); % dark canvas% Dpsi = 360.0 / 4.0 / 180.0 * pi; T = (Jmax − 1) / 2.0 / tan(Dpsi /2.0); DI = 0; DJ = 0; % ALPHA = 90.0; alpha = ALPHA / 180.0 * pi;sin_alpha = sin(alpha); cos_alpha = cos(alpha); % % ********* Right side******************************************* % Virtual screen Beta_o =45.0; for S = 0: 3 BETA = Beta_o + 360.0 * S / 4; beta = BETA / 180.0 *pi; sin_beta = sin(beta); cos_beta = cos(beta); % for I = 1: Imax for J= 1: Jmax p = J − Jo + DJ; q = I − Io + DI; theta = atan(sqrt(p{circumflex over ( )} 2 + q {circumflex over ( )} 2) / T); phi =atan2(q, p); % X = sin(theta) * cos(phi); Y = sin(theta) * sin(phi); Z =cos(theta); % Xp = X * cos_beta + Z * sin_beta; Yp = X * sin_alpha *sin_beta + Y * cos_alpha − Z * sin_alpha * cos_beta; Zp = −X *cos_alpha * sin_beta + Y * sin_alpha + Z * cos_alpha * cos_beta; %theta_p = acos(Zp); phi_p = atan2(Yp, Xp); r_p = gain * polyval(coeff,theta_p); y_p = Ko + r_p * sin(phi_p); x_p = Lo + r_p * cos(phi_p); Km =floor(y_p); Kp = Km + 1; dK = y_p − Km; Lm = floor(x_p); Lp = Lm + 1; dL= x_p − Lm; if((Km >= 1) & (Kp <= Kmax) & (Lm >= 1) & (Lp <= Lmax))EI(I, S * (Jmax + 1) + J, :) = (1 − dK) * (1 − dL) * CI(Km, Lm, :) ... +dK * (1 − dL) * CI(Kp, Lm, :) ... + (1 − dK) * dL * CI(Km, Lp, :) ... +dK * dL * CI(Kp, Lp, :); else EI(I, S * (Jmax + 1) + J, :) = zeros(1, 1,3); end end end % % ********* Lines******************************************* for I = 1: Imax EI(I, S *(Jmax + 1) + 1, :) = zeros(1, 1, 3); end end DI = uint8(EI − 1);imwrite(DI, ‘result.jpg’, ‘jpeg’); imagesc(DI); axis equal;

Ninth Embodiment

FIG. 60 is a schematic diagram of an exemplary image processing meansembodying the conception of the present invention. The image processingmeans(6016) of the present invention has an input frame buffer(6071)storing one frame of image acquired from the image acquisitionmeans(6010). If the image acquisition means(6010) is an analog CCTV,then it is necessary to decode NTSC, PAL, or Secam signals, and todeinterlace the two interlaced sub frames. On the other hand, thisprocedure is unnecessary for a digital camera. After these necessaryprocedures have been taken, the input frame buffer(6071) stores adigital image acquired from the image acquisition means(6010) in theform of a two dimensional array. This digital image is the uncorrectedimage plane. On the other hand, the output frame buffer(6073) stores anoutput signal in the form of a two dimensional array, which correspondsto the processed image plane(6035) that can be displayed on the imagedisplay means(6017). A central processing unit (CPU: 6075) furtherexists, which generates a processed image plane from the uncorrectedimage plane existing in the input frame buffer and stores in the outputframe buffer. The mapping relation between the output frame buffer andthe input frame buffer is stored in a non-volatile memory(6079) such asa SDRAM in the form of a lookup table. In other words, using thealgorithms from the embodiments of the current invention, a long list ofpixel addresses for the input frame buffer corresponding to particularpixels in the output frame buffer is generated and stored. Centralprocessing unit(6075) refers to this list stored in the nonvolatilememory in order to process the image.

Situation awareness module or an image selection means has beendescribed in the previous embodiments of the current invention. Theimage selection means(6077) in FIG. 60 receives the signals coming fromvarious sensors and switches connected to the imaging system and sendthem to the central processing unit. For example, by recognizing thebutton pressed by the user, it can dictate whether the originaldistorted fisheye image is displayed without any processing, or apanoramic image with a cylindrical or a Mercator projection scheme isdisplayed, or a rectilinear image is displayed. Said nonvolatile memorystores a number of list corresponding to the number of possible optionsa user can choose.

The image processing means of the embodiments of the current inventionare preferably implemented on a FPGA (Field Programmable Gate Array)chip. In this case, the said central processing unit, the said inputframe buffer, and the said output frame buffer can all be realized onthe FPGA chip.

In the case where FPGA chip is used to realize the image processingmeans, it is not impossible to directly implement the algorithm of theembodiments of the current invention. However, implementing seeminglysimple functions such as trigonometrical functions or division on FPGAchip is a quite challenging task, and requires a considerable′ resource.As has been described previously, a better alternative is to make anaccurate look-up table using the developed algorithm, store the look-uptable on a non-volatile memory, and do the image processing referring tothis look-up table.

On the other hand, if the user prefers to be able to continuously changethe pan or the tilt angle of rectilinear images, then it is impossibleto prepare all the corresponding look-up tables in advance. In thiscase, the central processing unit can dynamically generate the look-uptable referring to the algorithm and the parameters corresponding to theuser's selection, and store them on a volatile or a non-volatile memory.Therefore, if the user does not provide a new input by appropriate meanssuch as moving the mouse, then already generated look-up table is usedfor the image processing, and when a new input is supplied, then acorresponding look-up table is promptly generated and stored in the saidmemory.

In one preferable embodiment of the imaging system of the currentinvention, said image acquisition means is an analog CCTV equipped witha fisheye lens with more than 180° FOV, said image processing means isan independent device using a FPGA chip and storing the image processingalgorithm on a non-volatile memory as a look-up table, and the saidanalog CCTV and the said image processing means are connected by signaland power lines.

Or, said image acquisition means is a network camera equipped with afisheye lens with more than 180° FOV, said image processing means is anindependent device using a FPGA chip and storing the image processingalgorithm on a non-volatile memory as a look-up table, and theuncorrected image plane acquired by the said network camera is suppliedto the said image processing means by the Internet.

Preferred embodiments of the current invention have been described indetail referring to the accompanied drawings. However, the detaileddescription and the embodiments of the current invention are purely forillustrate purpose, and it will be apparent to those skilled in the artthat variations and modifications are possible without deviating fromthe sprits and the scopes of the present invention.

INDUSTRIAL APPLICABILITY

Such panoramic imaging systems and devices can be used not only insecurity surveillance applications for indoor and outdoor environments,but also in diverse areas such as video phone for apartment entrancedoor, rear view camera for vehicles, visual sensor for robots, and alsoit can be used to obtain panoramic photographs using a digital camera.

1. An imaging system comprising: an image acquisition means foracquiring an uncorrected image plane using a wide-angle lensrotationally symmetric about an optical axis, an image processing meansfor extracting a panoramic image plane from the uncorrected image plane;and an image display means for displaying the panoramic image plane on arectangular screen, wherein, if a coordinate of an image point on therectangular screen corresponding to an object point having a coordinate(X, Y, Z) in a world coordinate system, which has a nodal point of thewide-angle lens as an origin and a vertical line passing through theorigin as an Y-axis and an intersection line between a reference plane,which is a plane containing both the Y-axis and the optical axis of thewide-angle lens, and a horizontal plane perpendicular to the Y-axis as aZ-axis, is given as (x″, y″), the lateral coordinate x″ of the imagepoint is given as x″=cψ, and the longitudinal coordinate y″ of the imagepoint is given as y″=cF(δ), wherein, c is a proportionality constant, ψis a horizontal incidence angle an incident ray originating from theobject point makes with an Y-Z plane and is given as${\psi = {\tan^{- 1}\left( \frac{X}{Z} \right)}},$ d is a verticalincidence angle the incident ray makes with a X-Z plane and is given as${\delta = {\tan^{- 1}\left( \frac{Y}{\sqrt{X^{2} + Z^{2}}} \right)}},$and the function F(d) is a monotonically increasing function of thevertical incidence angle.
 2. The imaging system of claim 1, wherein thefunction F(d) of the vertical incidence angle d of the incident ray isgiven by any one among F(δ)=tan δ, F(δ)=δ and${F(\delta)} = {\ln {\left\{ {\tan\left( {\frac{\delta}{2} + \frac{\pi}{4}} \right)} \right\}.}}$3. An imaging system comprising: an image acquisition means foracquiring an uncorrected image plane using a fisheye lens rotationallysymmetric about an optical axis, an image processing means forextracting a panoramic image plane from the uncorrected image plane; andan image display means for displaying the panoramic image plane on arectangular screen, wherein, if a coordinate of an image point on therectangular screen corresponding to an object point having a coordinate(X, Y, Z) in a world coordinate system, which has a nodal point of thefisheye lens as an origin and a vertical line passing through the originas an Y-axis and an intersection line between a reference plane, whichis a plane containing both the Y-axis and the optical axis of thefisheye lens, and a horizontal plane perpendicular to the Y-axis as aZ-axis, is given as (x″, y″), the lateral coordinate x″ of the imagepoint is proportional to a horizontal incidence angle ψ, herein, thehorizontal incidence angle ψ is an angle an incident ray originatingfrom the object point makes with an Y-Z plane and is given as${\psi = {\tan^{- 1}\left( \frac{X}{Z} \right)}},$ and straight linesparallel to the Y-axis are captured as straight lines parallel to any″-axis on the screen.
 4. The imaging system of claim 3 furthercomprising an image selection means.
 5. The imaging system of claim 4,wherein the image selection means comprise a situation awareness module.6. The imaging system of claim 4, wherein the image selection meanscomprise a direction sensing means.
 7. An imaging system comprising: animage acquisition means for acquiring an uncorrected image plane using afisheye lens rotationally symmetric about an optical axis, an imageprocessing means for extracting a panoramic image plane from theuncorrected image plane; and an image display means for displaying thepanoramic image plane on a rectangular screen, wherein, if the opticalaxis of the fisheye lens is aligned parallel to a horizontal plane, anda coordinate of an image point on the rectangular screen correspondingto an object point having a coordinate (X, Y, Z) in a world coordinatesystem, which has a nodal point of the fisheye lens as an origin and avertical line passing through the origin as an Y-axis and the opticalaxis of the fisheye lens as a Z-axis, is given as (x″, y″), the lateralcoordinate x″ of the image point is proportional to a horizontalincidence angle ψ, herein, the horizontal incidence angle w is an anglean incident ray originating from the object point makes with an Y-Zplane and is given as ${\psi = {\tan^{- 1}\left( \frac{X}{Z} \right)}},$and the longitudinal coordinate y″ of the image point is given as amonotonically increasing function F(d) of a vertical incidence angle d,herein, the vertical incidence angle d is an angle the incident raymakes with a X-Z plane and is given as$\delta = {{\tan^{- 1}\left( \frac{Y}{\sqrt{X^{2} + Z^{2}}} \right)}.}$8. The imaging system of claim 7, wherein the function F(d) of thevertical incidence angle d of the incident ray is given by any one amongF(δ)=tan δ, F(δ)=δ and${F(\delta)} = {\ln {\left\{ {\tan\left( {\frac{\delta}{2} + \frac{\pi}{4}} \right)} \right\}.}}$9. An imaging system comprising: an image acquisition means foracquiring an uncorrected image plane using a fisheye lens rotationallysymmetric about an optical axis, an image processing means forextracting a panoramic image plane from the uncorrected image plane; andan image display means for displaying the panoramic image plane on arectangular screen, wherein, if the optical axis of the fisheye lens isaligned parallel to a horizontal plane, and a coordinate of an imagepoint on the rectangular screen corresponding to an object point havinga coordinate (X, Y, Z) in a world coordinate system, which has a nodalpoint of the fisheye lens as an origin and a vertical line passingthrough the origin as an Y-axis and the optical axis of the fisheye lensas a Z-axis, is given as (x″, y″), the lateral coordinate x″ of theimage point is proportional to a horizontal incidence angle ψ, herein,the horizontal incidence angle w is an angle an incident ray originatingfrom the object point makes with an Y-Z plane and is given as${\psi = {\tan^{- 1}\left( \frac{X}{Z} \right)}},$ and straight linesparallel to the Y-axis are captured as straight lines parallel to any″-axis on the screen.
 10. An imaging system comprising: an imageacquisition means for acquiring an uncorrected image plane using awide-angle lens rotationally symmetric about an optical axis, an imageprocessing means for extracting a rectilinear image plane from theuncorrected image plane; and an image display means for displaying therectilinear image plane on a rectangular screen, wherein, if acoordinate of an image point on the rectangular screen corresponding toan object point having a coordinate (X, Y, Z) in a world coordinatesystem, which has a nodal point of the wide-angle lens as an origin anda vertical line passing through the origin as an Y-axis, is given as(x″, y″), arbitrary straight lines in the world coordinate system appearas straight lines on the screen.
 11. The imaging system of claim 10,wherein a straight line parallel to the Y-axis in the world coordinatesystem and intersecting with a principal direction of vision for therectilinear image plane appears as a straight line parallel to any″-axis on the screen.
 12. The imaging system of claim 10 furthercomprising an image selection means for determining parameters of therectilinear image plane.
 13. The imaging system of claim 12, wherein theparameters of the rectilinear image plane comprise a size of theprocessed image plane (I_(max), J_(max)), a horizontal field of view Δψbefore any of the slide, pan and tilt operations, an amount of lateralslide operation ΔJ, an amount of longitudinal slide operation ΔI, a panangle, and a tilt angle.
 14. An imaging system comprising: an imageacquisition means for acquiring an uncorrected image plane using afisheye lens rotationally symmetric about an optical axis, an imageprocessing means for extracting a rectilinear image plane from theuncorrected image plane; and an image display means for displaying therectilinear image plane on a rectangular screen, wherein, if acoordinate of an image point on the rectangular screen corresponding toan object point having a coordinate (X, Y, Z) in a world coordinatesystem, which has a nodal point of the fisheye lens as an origin and avertical line passing through the origin as an Y-axis, is given as (x″,y″), arbitrary straight lines in the world coordinate system appear asstraight lines on the screen, a principal direction of vision for therectilinear image plane is different from the optical axis direction ofthe fisheye lens, and the principal direction of vision for therectilinear image plane does not pass through the center of therectilinear image plane.
 15. An imaging system comprising: an imageacquisition means for acquiring an uncorrected image plane using awide-angle lens rotationally symmetric about an optical axis, an imageprocessing means for extracting a rectilinear image plane from theuncorrected image plane; and an image display means for displaying therectilinear image plane on a rectangular screen, wherein, if acoordinate of an image point on the rectangular screen corresponding toan object point having a coordinate (X, Y, Z) in a world coordinatesystem, which has a nodal point of the wide-angle lens as an origin anda vertical line passing through the origin as an Y-axis, is given as(x″, y″), the rectilinear image plane is obtained from the uncorrectedimage plane using digital slide operation and at least one digital panor tilt operation, and arbitrary straight lines in the world coordinatesystem appear as straight lines on the screen.
 16. An imaging systemcomprising: an image acquisition means for acquiring an uncorrectedimage plane using a wide-angle lens rotationally symmetric about anoptical axis, an image processing means for extracting a multipleviewpoint panoramic image plane from the uncorrected image plane; and animage display means for displaying the multiple viewpoint panoramicimage plane on a rectangular screen, wherein, the multiple viewpointpanoramic image plane includes the 1^(st) through the n^(th) subrectilinear image planes laid out horizontally on the screen where n isa natural number larger than 2, an arbitrary straight line in a worldcoordinate system, which has a nodal point of the wide-angle lens as anorigin, appears as a straight line on any of the 1^(st) through then^(th) sub rectilinear image planes, and any straight line in the worldcoordinate system appearing on more than two adjacent sub rectilinearimage planes appears as connected line segments.
 17. The imaging systemof claim 16, wherein the 1^(st) through the n^(th) sub rectilinear imageplanes have identical magnification ratios, and the optical axis of thewide-angle lens penetrates one sub rectilinear image plane among the nsub rectilinear image planes.
 18. An imaging system comprising: an imageacquisition means for acquiring an uncorrected image plane using awide-angle lens rotationally symmetric about an optical axis, an imageprocessing means for extracting a processed image plane from theuncorrected image plane; and an image display means for displaying theprocessed image plane on a rectangular screen, wherein, the processedimage plane includes the 1^(st) through the n^(th) sub rectilinear imageplanes where n is a natural number larger than 2, an arbitrary straightline in a world coordinate system having a nodal point of the wide-anglelens as an origin appears as a straight line on any of the 1^(st)through the n^(th) sub rectilinear image planes.
 19. Imaging system ofclaim 18, wherein a straight line parallel to the Y-axis in the worldcoordinate system and intersecting with a principal direction of visionfor any sub rectilinear image plane appears as a straight line parallelto an y″-axis on the screen.
 20. The imaging system in claim 18 furthercomprising an image selection means for determining parameters of thesub rectilinear image planes.
 21. The imaging system of claim 18,wherein the 1^(st) through the n^(th) sub rectilinear image planes haveidentical magnification ratios.
 22. An imaging system comprising: animage acquisition means for acquiring an uncorrected image plane using awide-angle lens rotationally symmetric about an optical axis, an imageprocessing means for extracting a processed image plane from theuncorrected image plane, an image selection means for determiningprojection schemes and parameters of the processed image plane; and animage display means for displaying the processed image plane on arectangular screen, wherein, the projection schemes of the processedimage plane include a panoramic projection scheme and a rectilinearprojection scheme, if a coordinate of an image point on the rectangularscreen corresponding to an object point having a coordinate (X, Y, Z) ina world coordinate system, which has a nodal point of the wide-anglelens as an origin and a vertical line passing through the origin as anY-axis, is given as (x″, y″), the panoramic projection scheme is ascheme wherein the lateral coordinate x″ of the image point isproportional to a horizontal incidence angle ψ, herein, the horizontalincidence angle ψ is an angle an incident ray originating from theobject point makes with an Y-Z plane and is given as${\psi = {\tan^{- 1}\left( \frac{X}{Z} \right)}},$ and straight linesparallel to the Y-axis are captured as straight lines parallel to any″-axis on the screen, and the rectilinear projection scheme is a schemewherein arbitrary straight lines in the world coordinate system appearas straight lines on the screen.
 23. The imaging system of claim 22,wherein the wide-angle lens is a fisheye lens.
 24. The imaging system ofclaim 22, wherein the image selection means comprise a situationawareness module.
 25. The imaging system of claim 22, wherein the imageselection means comprise a direction sensing means.
 26. The imagingsystem of claim 22, wherein a straight line parallel to the Y-axis inthe world coordinate system and intersecting with a principal directionof vision for a processed image plane having a rectilinear projectionscheme appears as a straight line parallel to the y″-axis on the screen.27. The imaging system of claim 22, wherein the image selection meanscomprise a direction sensing means, and the image acquisition means is avideo camera installed on an automobile.
 28. The imaging system of claim22, wherein the image acquisition means is a camera module installed ona camera phone, the image processing means is embedded software runningin the camera phone, and the image display means is a display screen ofthe camera phone.
 29. The imaging system of claim 22, wherein the imageacquisition means is a camera module installed on a camera phone, theimage processing means is software running on a computer, and the imagedisplay means is a display monitor of the computer.
 30. The imagingsystem of claim 22, wherein the image acquisition means is a PC camera,the image processing means is software running on a computer, and theimage display means is a display monitor of the computer.
 31. Theimaging system of claim 22, wherein the image acquisition means is adigital camera, the image processing means is software running on acomputer, and the image display means is a display monitor of thecomputer.
 32. The imaging system of claim 22, wherein the imageacquisition means is a camera module in a network camera, and the imageprocessing means is embedded software running in the network camera. 33.The imaging system of claim 22, wherein the image acquisition means is acamera module on a network camera, the image processing means issoftware running on a computer connected to the same network as thenetwork camera, and the image display means is a display monitor of thecomputer.
 34. The imaging system of claim 22, wherein the imageacquisition means is a camera module on a video doorphone, and the imageprocessing means is embedded software running in the video doorphone,and the image display means is a wallpad monitor in the house.
 35. Theimaging system of claim 22, wherein the image acquisition means is acamera module on a video doorphone, and the image processing means isembedded software running in the video doorphone, and the image displaymeans is display monitor of a smartphone connected to the videodoorphone using a wireless Internet.
 36. The imaging system of claim 22,wherein the image acquisition means is a camera module on a broadcastingcamera, and the image processing means is embedded software running inthe broadcasting camera.